Diagnostic instrumentation

ABSTRACT

Diagnostic Instrumentation system can use diagnostic monitors. The diagnostic monitors can be indicated using instrumentation at class loading time. The diagnostic monitors can have associated diagnostic functions which can be enabled or disenabled while the server software is running.

CLAIMS OF PRIORTY

This application claims priority to U.S. Provisional Application No.60/573,280 entitled “Diagnostics” filed May 21, 2004 by Shrivastava etal.

BACKGROUND OF INVENTION

Server software, such as application servers, can provide applicationinfrastructure for modern business systems. Application servers cansupport applications that share data and resources with other systems,and can generate dynamic information for Web pages and other userinterfaces. The application servers can provide a layer of functions andservices between Web servers and underlying applications and databases.As companies deploy large applications to support secure transactionsand use Web and XML protocols, the growing complexity of theapplications can require a managed environment for the applications.Globally deployed applications with complex business logic potentiallyservicing tens of thousands of concurrent users in real-time, require ascalability and reliability that makes quality application servers anecessity.

Application Servers can provide functions and services, such as loadbalancing, fault tolerance, web services, network transparency, legacyintegration, transaction management, security, messaging,multi-threading, persistence, database connectivity, resource pooling,and development, testing, and packaging facilities. Built-infunctionality from the application server can speed applicationdevelopment and can relieve developers of the effort and expense ofcreating important services on their own. Applications can then providemore functionality, be developed faster, be less costly to maintain, andprovide a greater return on investment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a server software system including a diagnosticframework of one embodiment of the present invention.

FIG. 2 is a diagram of a diagnostic framework of one embodiment of thepresent invention.

FIG. 3A is a diagram illustrating a work area protocol unit in a serversoftware.

FIG. 3B is a conceptual diagram illustrating the diagnostic contextassociated with a request.

FIG. 3C is a flow chart illustrating one embodiment of a method of thepresent invention.

FIG. 3D is a diagram of another embodiment of the present invention.

FIG. 4A illustrates the use of instrumentation in one embodiment of thepresent invention.

FIG. 4B illustrates the use of the diagnostic monitor of one embodimentof the present invention.

FIG. 5 illustrates a method of one embodiment of the present invention.

FIG. 6A illustrates the operation of a metadata-based selection of debuginformation of one embodiment.

FIG. 6B illustrates a debugscope tree for metadata of one embodiment ofthe present invention.

FIGS. 7A and 7B illustrate methods of embodiments of the presentinvention.

DETAILED DESCRIPTION

A number of embodiments of the present invention concern inventions thatcan be used as part of a diagnostic framework. A diagnostic frameworkcan be used to monitor server software and applications deployed to it.The diagnostic framework can a part of or separate from the serversoftware.

FIG. 1 illustrates a diagnostic framework 102 for server software 104.The server software can be, for example, a J2EE-based server software,such as the WebLogic server available from BEA Systems, Inc., of SanJose, Calif. Typically, the server software 104 interacts with a backendtier 106 to provide services for clients, such as clients 108 and 110.The diagnostic framework 102 can interact with a third party analysissoftware 112. The diagnostic framework can provide access to thediagnostics for the server software and for applications that run on theserver software.

FIG. 2 shows a non-limiting example of a diagnostic framework 200. Asshown in the example of FIG. 2, a Diagnostic Framework 200 can consistof a number of components that work together to collect, archive, andaccess diagnostic information about the server and the applications ithosts.

Data Creators 202 can generate diagnostic data that is consumed by aCollector 204 The Collector 204 can coordinate with an Archive 206 topersist this data and with the Watch & Notification system 208 toprovide automated monitoring. The Accessor 210 can interact with theCollector 204 to expose current diagnostic data and with the Archive 206to present historical data. The Manager 212 can provide a configurationand control interface for managing the diagnostic framework 200. TheImage Capture facility 214 can provide a model for capturing adiagnostic snapshot of key server state.

In one embodiment, the framework components operate at the server leveland are only aware of server scope. In one embodiment, the componentsexcept for the Manager can exist entirely within the server process andparticipate in the standard server lifecycle. In one embodiment,artifacts of the framework can be configured and stored on a per serverbasis.

Diagnostic data can be collected from a number of sources. These sourcescan be logically classified as either providers 202 a, data creatorsthat are sampled at regular intervals to harvest current values, orpublishers 202 b, data creators that synchronously generate events.

In one embodiment, within the server codebase, invocations of thelogging infrastructure through the catalog infrastructure, the debuggingmodel, or directly through the logger serve as inline data publishersand the generated data is collected as events.

The instrumentation system 216 can create monitors and advice, some ofwhich are publishers and some of which are providers, and insinuate themat well defined points in the flow of execution. Publishers 202 b cangenerate events that are consumed by a traditional logging framework 204a. Providers 202 a can expose their data to the Harvester 204 b using adata source interface. In one embodiment, components registered with theMBean Server may also make themselves known as data providers byregistering with the Harvester 204 b.

Providers 202 a registered with the Harvester 204 b can be eligible forcollection based on the current harvesting configuration, which isdynamically controllable through the management interface. Collecteddata can then be exposed to both the Watch & Notification system 108 forautomated monitoring and to the Archive 206 for persistence.

Past state is often important to diagnostic faults in a system. Statecan be captured and archived for future access, creating a historicalarchive. In the diagnostic framework 200, the Archive 206 can meet thisneed using persistence components. Both events and harvested metrics canbe persisted and made available for historical review.

Traditional logging information, which is human readable and intendedfor inclusion in the server log can be persisted through the standardlogging appenders 206 a. New event data that is intended for systemconsumption and may be compacted and need not be human readable can bepersisted into an event store 218 using an event archiver 206 b. Metricdata can be persisted into a data store 220 using a data archiver 206 c.

The archive 106 can provide access interfaces so that the Accessor 110may expose any of the persisted historical data.

The Watch & Notification system 208 can be used to create automatedmonitors to look for specific diagnostic states and send notificationbased on configured rules.

A watch rule 208 a can monitor either event data from a Data Publisher202 b or metric data from a Data Provider 202 a that is harvested by theHarvester. The Watcher 208 can be capable of managing watches 202 b thatare composed of a number of watch rules. For each watch 202 b, a numberof notifications 202 c may be configured. By default every watch 202 bcan log an event in the server log. In addition, an number ofnotifications, such as Simple Mail Transfer Protocol (SMTP), SimpleNetwork Management Protocol (SNMP), Java Management extensions (JMX),and Java Messaging Service (JMS) notifications, can be supported.

The Accessor 210 can provide access to the data collected by theframework 200 including event and metric data. The Accessor 210 caninteract with the Harvester 204 b to get the current state ofharvestable values in the server. The Accessor 210 can interact with theArchive 206 to get historical state including logged event data andpersisted metrics.

When accessing data in a running server, a JMX based access service canbe used. The Accessor 210 can provide for data lookup by type, bycomponent, and by attribute. It permits time based filtering and in thecase of events filtering by severity, source and content.

Tools may wish to access data that was persisted by a server which isnot currently active. In these cases, an offline Accessor can beprovided. The offline Accessor can support access to historical data.

The Manager 212 can provide a management interface for the diagnosticframework 200. The Manager 212 can provide access to a server specifictop level Diagnostic Service 212 a which then provides references to theappropriate component specific services. Each service can then referencethe appropriate configuration component and the appropriate runtimecomponent running in the content of the server of interest.

The Manager 212 can provide the management interface of the entireframework. It can provide access to a server specific top levelDiagnostic Service 212 a which then provides references to theappropriate component specific services. Each service can then referencethe appropriate configuration component and the appropriate runtimecomponent running in the content of the server of interest.

The management interface can be JMX-based and use service orientedinterfaces. The service components can be defined as Runtime MBeans. Theservice components can exist in the administration server and interactwith runtime components that exist in the server process associated withthe framework instance. In addition to management services, theDiagnostic Service can provide diagnostic data about the frameworkitself and a reference to the Accessor for the given server.

The Diagnostic Image support 214 can gather the most common sources ofkey server state used in diagnosing problems and packages that stateinto a single artifact that can be made available to support in the nearterm and may be fed into automated analysis tools in the future. Thediagnostic image can be in essence, a diagnostic snapshot or dump fromthe server. The image capture support 214 can include both an on demandcapture process and an automated capture based on some basic failuredetection.

In one embodiment, a diagnostic framework can capture and archivemeaningful state that may be used to monitor and diagnose problems thatarise in a running server. In one embodiment, the framework is focusedon capturing and providing diagnostic data but does not require any ofconsumption tools that may access, analyze, or present the data.

The diagnostic framework can pull all diagnostics features andfunctionality under one umbrella, and can be flexible enough to supportmodules such as system analysis and feedback control required foradaptive systems management

The framework can expose activation and control of all key featuresdynamically in a running environment, providing the most significantdiagnostic value in the state which much be investigated.

The diagnostic framework can introduce a minimal inactive overhead and areasonable overhead when significant diagnostic collection is in effect.The diagnostic framework can allow for expansion for both the data andfunctionality available in the core offering, and allow applicationdevelopers and tools vendors flexibility to easily integrate and extendthe framework with advanced tools.

Diagnostic Context

One embodiment of the present invention comprises receiving a requestinto server software; associating a diagnostic context with the request,and at a later diagnosis point of the server software, using thediagnostic context to determine whether to do a diagnosis function.

FIG. 3A illustrates an example when a request 302 is received by serversoftware 304. The request typically causes a number of differentoperations by the server software 304. The diagnostic context can allowdiagnostic information to be collected for a specific request or type ofrequest. In one embodiment, the diagnostic context can be implementedusing a work area protocol. An example of a work area protocol is givenin the Java Service Request (JSR) 149. The work area protocol describesthe association of a context with requests. This context can be providedby the work area protocol across different servers and for the life ofthe request. In the example of FIG. 3A, a work area protocol unit 306 isused by the server software 304 associate contexts with requests.

FIG. 3B illustrates a conceptual diagram of a request 310 having adefined work area context 312. All or a portion of the work area context312 can be a diagnostic context 314. The diagnostic context 314 can beused by the server software to determine whether to do a diagnosticfunction. In one embodiment, a dye vector 316 is used to determinewhether to do the diagnostic function.

By using a diagnostic context associated with a request to determinewhether to do diagnostic functions such as collecting diagnostic data,the amount of diagnostic data can be reduced preventing theadministrator from being flooded with data. In one embodiment, onlydiagnostic data associated with the request is collected.

FIG. 3C illustrates a flow chart of one embodiment of a method of thepresent invention. In step 320, a request is received by the serversoftware. In step 322, a diagnostic context is associated with therequest. This can be done by using the work area protocol. Later, instep 324, the system can come to a diagnosis point. In step 326, thediagnostic context can be used to determine whether to do a diagnosticfunction. In one embodiment, a mask can be compared to a dye vector ofthe diagnostic context. In step 328, based upon the comparison, thediagnostic function is either done in step 330 or not done in step 332.In one embodiment, the dye vector is ANDed with the mask and thencompared to the mask to determine whether to do the diagnostic function.The diagnostic function can be any diagnostic operation including usinga counter to determine how many times the request goes thorough thediagnosis point, storing arguments, and the like. The diagnosticfunction can be a part of a diagnostic monitor as described below.

Proper functioning of an application server can be monitored bycollecting and analyzing diagnostic data while the server is running.However, in a production environment, large number of requests istypically serviced by the server, which can result in informationoverload. Generation, collection and analysis of diagnostic data canimpose heavy burden on the server as well as analysis tools/personnel,which in turn can result in general slowdown.

A Request Dyeing feature can make it possible to focus on only requestsof interest, without adverse impact on the performance of rest of theserver. It also makes it possible to keep the volume of diagnostic datato manageable levels, thus facilitating analysis of diagnostic data.

One embodiment of the present invention is software comprising code toassociate a diagnostic context with a request into server software; andcode to use the diagnostic context to determine whether to do adiagnosis function at a later diagnosis point of the server software.

The code to associate the diagnostic context with a request into theserver software can be code associated with a work area protocol. Thecode to use the diagnostic context can be the code that obtains thediagnostic context compares the dye vector with a mask and thendetermines whether to do the diagnosis function.

FIG. 3D illustrates another use of a diagnostic context. In step 340, arequest is received. In step 342, a diagnostic context is associatedwith the request. In step 344, the diagnostic context for the request isstored along with the diagnosing data for the request. For an example,diagnostic context ID identifying the request can be stored along withthe diagnostic data so that later functions can easily obtain thediagnostic data for the request from an archive for example.

Storing the diagnostic context ID along with the diagnostic contextinformation makes it easy to track the diagnosis information associatedwith a request. Tools can be used to get the stored diagnosisinformation of the request.

One embodiment of the present invention comprises receiving a requestinto server software; associating a diagnostic context with the request;and storing the diagnostic context information for the request alongwith the diagnosis data for the request.

A busy production server can generate large amount of diagnostic data.As requests flow through the system, they can generate diagnosticinformation. Often, it is useful to analyze this related information. ARequest Correlation feature provides a means to correlate diagnosticinformation which is generated as a request progresses through thesystem.

Diagnostic Instrumentation

One embodiment of the present invention comprises instramenting classesof server software with a diagnostic monitor, the diagnostic monitorincluding at least one diagnostic function; and enabling the diagnosticfunction at the diagnostic monitor while the server software is runningwithout restarting the server software.

FIG. 4A illustrates an embodiment illustrating the instrumentation ofclasses in one embodiment. Instrumentation can be done for diagnosticmonitors to include a diagnostic action (diagnostic function).Typically, the instrumentation can be done when a class is loaded. Ifthe Java Virtual Machine (JVM) is modified, the instrumentation can bedone during runtime using hot swap functionality enabled by some virtualmachines such as JRockit. One embodiment of the present invention,describes a way of allowing for the modification, addition and removalof diagnostic functions from the system without requiring a JVM thatprovides hot swap functionality.

As shown in FIG. 4B, pointcut can include a set of joint points. When aprogram reaches one of the joint points described by the pointcut, code(such as an advice) associated with the pointcut is executed. Pointcutsindicate where additional code on top of the clearly defined behaviorcan be inserted.

In one embodiment, a diagnostic monitor 420 is associated with apointcut, thus in the instrumented class each of the joint points canhave an indicator to the diagnostic monitor 420. In one embodiment, theadvises for the diagnostic functions can include a name, type, dye maskand properties. Runtime Management software 424 can be used to enable adiagnostic monitor or to modify a list of associated advices for thediagnostic monitor. Diagnostic functions can be added or removed duringruntime by a runtime management unit 424. This can be done by adding orremoving an advice from the list of advices for the diagnostic monitor.Diagnostic monitors that can have the diagnostic functions or adviceschanged while the server is running can be called delegating monitors.

A dye mask can be used so that the diagnostic function is run only forsome subset of the requests as indicated by the dye vector in thediagnostic context for the request. In one embodiment, removing all ofthe listed advises from the diagnostic monitor 420 disables thediagnostic functions associated with the diagnostic monitor.

The diagnostic functions can include counters, and the saving ofarguments, including the saving of input arguments and the saving ofoutput arguments. In one embodiment, the diagnostic functions arelimited by the server software to a predetermined set of functions. Theserver software can limit the diagnostic functions to a predeterminedset of actions to prevent the functionality of the server software frombeing overly modified.

The instrumentation can include weaving in an indication of thediagnostic monitor at each joint point. A library of diagnostic monitorscan be provided along with a variety of advices. Note that thediagnostic functions can be selected for diagnostic monitor while theserver software is running without restarting the server software,because although the instrumentation is done at class loading time, thediagnostic monitor 420 can be modified by the management unit 424without requiring a restart of the server software.

One embodiment of the present invention is software comprising aninstrumentation unit to instrument server software at a pointcut with adiagnostic monitor. The diagnostic monitor having at least onediagnostic function. The diagnostic function can be enabled for apointcut at the diagnostic monitor while the server software is runningwithout restarting the server software.

One embodiment of the present invention is a method comprisinginstramenting classes of server software with a diagnostic monitor; andselecting and enabling a diagnostic function for the diagnostic monitorwhile the server software is running without restarting the serversoftware. Such diagnostic monitors can be called delegating monitors.

Visibility into functioning of a running application is important fordiagnosing problems and identifying bottlenecks before they becomecritical. The application programmer can provide visibility by addingsuitable diagnostic code into the application. However, it is tediousand not flexible, since it cannot be changed after the application isdeveloped. Also, it can introduce some performance penalty even thoughnot in use.

A Dynamic Declarative Diagnostic instrumentation system provides adynamic and declarative way of adding diagnostic code into already builtapplications. Such code can be added with no programming effort by theapplication developer or the administrator. The diagnostic code can bedynamically added or removed from the application.

Diagnostic Image

One embodiment of the present invention comprises upon an event,compiling diagnostic information about the state of server software intoa data stream; and compressing the data stream.

FIG. 5 illustrates an embodiment of the present invention. In step 502,the diagnostic data is collected. Typically, the diagnostic data is keptin volatile memory. The diagnostic data can be collected at differentcollection points through the server software and applications. In Step504, upon an event, diagnostic image is created. The event can berecognition by the server software of a prefailure state or can be anevent such as the user selecting the collection of the diagnostic image.The collected diagnostic information can be a diagnostic image includinginformation concerning multiple software components of the serversoftware. The diagnostic image file can be very large, for example, over1 megabyte. In step 506, the diagnostic image is compressed. In oneembodiment, the diagnostic information is compressed into a zip file.The compressed zip file can be sent across the internet from a serversoftware site to a central location, such as the office of the serversoftware vendor. The compressed diagnostic image can be sent in anemail.

In one embodiment, a diagnostic image is not created for each failure.The creation of multiple diagnostic images can be expensive because oflarge size of the diagnostic images. A timer can be used to inhibit thecollection of multiple diagnostic images.

One embodiment of the present invention includes software including codeto, upon an event, compile diagnostic information about the state ofserver software into a data stream; and code to compress the datastream.

Hierachical Debug

One embodiment of the present invention comprises server softwareincluding: debug points capable of activation to get debug information;and metadata identifying logical groupings of debug points to allowactivation of a group of debug points, wherein the metadata allows forthe activation of logically grouped debug points across packageboundaries.

The debug point can be a point at which a diagnosis function is done.The debug point can include a comparison to a request dye so that debuginformation is only collected for certain requests.

One embodiment of the present invention includes metadata basedselection of debug points. In one embodiment, the activation of thedebug points is done based upon stored debug activation selections. Oneway to do this is to use an MBean such as a service debug MBean toindicate which debug points are activated. MBean can be very complicatedfor large application servers and it can be hard for a user to selectthe debug points using the MBean.

Similar functions can be distributed within different packages. Forexample, security function can be distributed among various packages andif a person wants to debug security, they need to know exactly whichpackages to turn on for security. For example, in the Log4jinstrumentation system debug locations are turned on based upon thegranularity of the package names and the users need to know which of thepackages are associated with a given function.

FIG. 6A illustrates a case in which a metadata-based selection of adebug is done. A unit 602 is used for metadata-based selection of debug.This can be used to allow a user to update a debug storage 604 such as aserver debug MBean. The debug selection storage is then used for theactivation of debug 606 within the server software. In one embodiment, adiagnostic monitor for example can be activated based upon theinformation stored in the server debug MBean.

FIG. 6B illustrates one example of metadata that allows for theactivation of the debug points cross package boundaries. FIG. 6B showsdebugscope tree metadata. In this embodiment, a debugscope tree is usedto determine which attributes to store within the debug selectionstorage, such as which attributes to set in a debug MBean. In thisexample, the server MBean attributes are associated with nodes, such asleaves, of the debugscope tree. In this case, the user by enabling adebug for “WebLogic.EJB.security” can set the server debug MBean atAttribute A. If the user selects the MBean for “Weblogic.Security” theMBean attributes to the “Weblogic.Security” node can be set. In theexample of FIG. 6B, setting server MBean attributes A, B and C.

Metadata can be selected by a user from previously defined metadata. Inone embodiment, the metadata can be defined in an object such as aMBean.

In one embodiment, the debugs points can be turned on without restartingthe server software.

FIG. 6B illustrates a case where the metadata is arranged using ahierarchical arrangement of nodes. In one embodiment, when a parent nodeis enabled all of the attributes associated with child nodes can beenabled as well. A debug point can be associated with more than one nodeof the debugscope tree.

One embodiment of the present invention are methods including usingmetadata to activate a group of debug points on server software; andcollecting debug information from the debug points. The metadataidentifying logical groupings of debug points to allow activation of thegroup of debug points. The metadata allows for the activation oflogically grouped debug points across package boundaries.

A debugging system can use a declarative model. In one example, all thedebug settings are known at compile time along with its documentation asopposed to APIs like Java Logging & Log4j where it is known only atruntime. Hierarchical control can be done using configurable DebugScopes.

Watches and Notifications

One embodiment of the present invention is a method comprising:obtaining diagnosis information; using configured watch rules todetermine whether to produce a notification, the stored watch rulesbeing evaluated against the diagnosis information; and using configurednotification types to determine the type of notification to be sent.

FIG. 7A illustrates a method of one embodiment of the present invention.In step 702 diagnostic information is obtained. The diagnosticinformation can be debug information or the like. In step 704, aconfigured watch ruler is used to determine whether to produce anotification. The configured watch rules can be used to examine thediagnosis information to check whether the watch rule is matched. Ifthere is a match, a configured notification types can be used todetermine the type of notification to be sent in step 706. Thediagnostic information can be stored in memory such as in the volatilememory of a server software system. The watch rules can include be acompound rule including a number of primitive rules. The watch rules canbe analogous to SQL searches. The notification catch can include emailnotifications, Java Messaging Service (JMS) notifications, and messagesto queues, SMNP traps, log records, or the creation of a diagnosticimage. Alternately the notification types can include a JMXnotification. The watch rules can be stored in a file such as aconfig.XML file.

One embodiment of the present invention includes software comprisingcode to use configured watch rules to determine whether to produce anotification, the stored watch rules being evaluated against diagnosisinformation; and code to use a configured notification type to determinethe type of notification to be sent.

One embodiment of the present invention is a method including a methodcomprising: obtaining diagnosis information; persisting the diagnosisinformation into at least two different types of archives; using anaccessor component to interrogate the persisted diagnosis information inthe at least two different types of archives according to watch rules.

FIG. 7B illustrates an example of one embodiment of the presentinvention including the steps of obtaining diagnosis information in step710; persisting the diagnostic information to these two different typeof archives in step 712; and, using an accessor of component to obtainthe diagnosis information from at least of two different types ofarchives according to the watch rules, in step 714. The archives caninclude a database and flat files in which to log info or otherinformation watch is stored. Looking again at FIG. 2, the serveraccessor 210 can access the archive 202 for different types of archivestorage.

One embodiment of the present invention comprises software comprisingcode to persist diagnosis information into at least two different typesof archives; and code to interrogate the persisted diagnosis informationin the at least two different types of archives according to watchrules.

DETAILED DESCRIPTION OF ONE NON-LIMITING EXAMPLE

The following description gives one non-limiting implementation of adiagnostic framework. The discussion below gives one embodiment, butthose skilled in the art will understand that other implementations ofthe above concepts can be done. Any potentially limitating languagegiven below is to be interpreted in the context of the specificnon-limiting implementation and is not meant to limit the generalconcepts.

Non-Limiting Example of Diagnostic Context

A Diagnostic Framework can provide a unified framework to gainvisibility into the functioning of the server as well as applicationsrunning on it. It can consume, collate, archive and publish generateddiagnostic information. In order to correlate different pieces ofgenerated information, the generated diagnostics can capture contextualinformation. The contextual information can be maintained with respectto requests serviced by the Server. A request may cross thread and/orJava Virtual Machine (JVM) boundaries. A diagnostic context (DC) cancapture required contextual information for the request. The diagnosticcontext can be created when a request enters the system and can bepropagated to participating threads and virtual machines (VMs) as therequest crosses thread or VM boundaries. Such context propagation can behandled by the underlying work area protocol machinery of implementationof the server software WLS. The diagnostic context can be available tovarious diagnostic modules, such as logging, tracing, etc so that theycan record the contextual information. As requests enter the system andbeing serviced, different facilities provided by the DiagnosticFramework can generate diagnostic data, which can provide visibilityinto the working of the system. For example, messages may be loggedusing the logging system, or various diagnostic monitors woven into thesystem may record information as the request progresses. This activityis likely to generate volumes of information, which can be difficult todigest. For example, how does one correlate different log messages whichwere generated as part of the same request and isolate them frommessages corresponding to other requests. Moreover, in a multi-threadedand distributed environment, requests may cross thread and VMboundaries. It can be important to correlate diagnostic informationrelated to a request in that situation as well.

The diagnostic framework can use contextual information related torequests in the form of Diagnostic Context (DC). The Diagnostic Contextcan be made available to different facilities within diagnosticframework. These facilities such as logging, diagnostic monitors etc cantag generated information with contents of the diagnostic context. Thus,diagnostic information generated by them can be collated, filtered andcorrelated by external tools in a better way.

The diagnostic context can also make it possible to generate diagnosticinformation only when contextual information in the Diagnostic Contextsatisfies certain criteria. Thus, it can make it possible to keep thevolume of generated information to manageable levels and keep theoverheads of generating such information relatively low.

The machinery for creation and propagation of Diagnostic Context acrossthreads and VMs can be addressed by the work area protocol. Thediagnostic framework can participate in the initial creation andpopulation of Diagnostic Context using facilities provided for the workarea protocol. Thereafter, it can access the Diagnostic Context asneeded using facilities provided for the work area protocol.

The Diagnostic Context can capture request specific information andprovide an interface to access it. In one embodiment, it does notattempt to capture information which is already available elsewhere,such as user identity, transaction IDs, timestamps, etc.

As requests are serviced within the Server, different facilities withinthe diagnostic framework such as logging, diagnostic monitors etc mayrecord the progress of the requests. To facilitate correlation of suchinformation, these facilities can use contextual information related tothe requests. For example, a unique context identifier for a requestwould be useful to correlate events and log entries generated as part ofthe same request. Apart from the context identifier, other informationsuch as address of the originating client and some user definable flagswould be useful to decide if certain diagnostic information needs to begenerated at all.

The diagnostic context can be identified by a fixed name (such asweblogic.management.DiagnosticContext). Thus, it can be looked up usingthis name. Actual contextual data for the Diagnostic Context can bestored in the diagnostic context payload. The server software canperform context propagation with several protocols, such as HTTP, RMI,SOAP, T3, etc. The server software can also provide mechanism toidentify and access a context using a key. The diagnostic framework cancapture necessary contextual data needed by it. The diagnostic frameworkmay also need other pieces of information such as identity of associateduser, transaction identity, thread ID, timestamp, etc, while generatingdiagnostic information. These entities can be obtained from otherexisting sources. Therefore, the diagnostic context data need notduplicate it.

Propagation of diagnostic context can be potentially expensive.Therefore, it should be kept as compact as possible.

Diagnostic context can be created and populated when a request entersthe system by the server software. Upon request entry, the serversoftware can examine if a Diagnostic Context is included in the request.If so, it can be propagated. The server software can handle contextpropagation with several different protocols as mentioned above. If theDiagnostic Context is not included in the request, it can create a newcontext with specific name (such asweblogic.management.DiagnosticContext). The server software caninstantiate a Diagnostic Context payload, which implements theDiagContext interface described later. The created Diagnostic Contextcan be populated with the Diagnostic Context payload. Thus, within thescope of a request execution, existence of the Diagnostic Context can beguaranteed.

In one embodiment, the Diagnostic Context is not created and initializedon request entry when diagnostic monitoring is not enabled.

The Diagnostic Context can capture necessary contextual information fora request during the life-cycle of a request. It can be created andinitialized when a request enters the system, for example, when a clientmakes an HTTP request. Once created, diagnostic framework can retrievethe diagnostic context data from server software as needed. For example,the logging system can use this information while creating log recordsso that log entries for the same request can be correlated. Since thediagnostic context can be propagated as a request crosses thread and VMboundaries, it should be kept as small as possible. It can maintain thefollowing information:

-   -   Context Id: This can be a unique identifier with which a request        can be uniquely identified. It should be unique over the entire        domain.    -   A bit vector of dye flags, which can be turned on or off based        on certain conditions. Typically, they can be conditionally        turned on by diagnostic monitors when a request enters the        system. The business logic provided by advices associated with        the monitors can dictate the conditions under which the dye        flags can be turned on. Thereafter, other diagnostic monitors        can inspect the dye setting and conditionally perform actions,        such as generate events. The dye vector can contain 64 dye        flags. Each dye flag can be independently read and written to.

In one embodiment, a Diagnostic Context is guaranteed to exist withinthe scope of a request execution. It can be accessed and used bydifferent facilities such as logging and tracing systems. For example,log entries can emit the unique context identifier assigned to therequest along with other data for easy correlation. Accessing contextdata can involve:

-   -   Obtain the Diagnostic Context from underlying work area protocol        implementation, using its specific name        (weblogic.management.DiagnosticContext).    -   Obtain the context payload from the context.    -   Access and use context payload attributes, using methods        described in the DiagContext interface.

Requests can be dyed or specially marked to indicate that they are ofspecial interest. For example, in a running system, it may be desirableto send a specially marked test request, which can be conditionallytraced by the tracing monitors. This can allow creation of highlyfocused diagnostic information without slowing down other requests. Therequests can be typically marked when they enter the system by settingflags in the Diagnostic Context. The Diagnostic Context can provide anumber of flags (such as 64), which can be independently set or reset.Certain dyeing monitors can turn on these flags when the request entersthe system.

Thereafter, other diagnostic monitors can make use of these flags (dyes)to conditionally execute certain actions. For example, tracing can beenabled if the tracing dye is present in the request.

Some dye bits can be made available to be used by applications. Allother dye flags can be reserved for use by server software andlibraries.

Pointcuts which identify the places where the diagnostic contextcreated, should be defined to cover inbound requests. Following are somesuch places:

-   -   Inbound RMI calls    -   HTTP/HTTPS requests    -   Inbound t3 requests    -   Inbound connector requests, where work gets scheduled.        Following are exemplary criteria, which can be used to inject        dyes into the DC at request entry:    -   Client IP address    -   Client user identity    -   Cookies in HTTP headers (for HTTP only)    -   Protocol based dyeing    -   Execute queues    -   Thread group

Different dye flags can be reserved to identify that a request satisfieson or more criteria described above. A special diagnostic monitorDyeInjectionMonitor can be provided to inject specific dyes in thecontext based on configured parameters.

Non-Limiting Example of Diagnostic Image

This details the functional specification of the Diagnostic Imagefacility of a Diagnostic Framework. It includes both the model fordefining and registering important sources of diagnostic information andthe specific mechanisms for capturing an image.

The diagnosis of almost all faults in the server begins with a fewcommon diagnostic artifacts including the configuration file and thelogfile. This is evidenced by the current support protocol of manuallycollecting these items from customers at the inception of most supportcalls. To improve both customers' total cost of ownership and tostandardize and enhance the data collection process the DiagnosticFramework provides a Diagnostic Image facility. The Diagnostic Imagesupport can gather the most common sources of server state used indiagnosing problems and packages that state into a single artifact thatcan be made available to support and may be fed into automated analysis.Further, the Diagnostic Image support can include both an on demandcapture process, similar to the current manual process, and an automatedcapture based on some basic failure detection.

A Diagnostic Image can be a heavyweight artifact meant to serve as aserver level state dump for the purposes of diagnosing significantfailures. It provides on-the-spot data capture opportunities for anadministrator who can be asked to provide state data to server softwarecustomer support. For regular system monitoring and localized faultdetection, the standard data creators, harvester, and accessorcomponents can be utilized. The Diagnostic Image is a bulky data setwith no fine grained control or filtering support and creation of animage should not be a regular occurrence in most deployments.

Diagnostic Image management can be coordinated by a singleton ImageManager. For a component to participate in the Diagnostic Image captureprocess it must be can be defined as an Image Source and be registeredwith an Image Manager. A Diagnostic Image can be created by invoking thecapture mechanism exposed on the Image Manager. Automated capture can bemade possible by a basic First Failure detection mechanism. The ImageManager can be the singular and central resource for all DiagnosticImage handling. It can be a singleton in each process in which it isutilized. The Image Manager can provides additional functionality forImage Source registration and image capture as defined in subsequentsections of this document.

The Image Manager can be a singleton which is initialized on firstreference. The ImageManager can provide a static accessor to acquire areference to the singleton instance. In nearly all instances this can bewhen the first Image Source attempts to register during server startupas defined in the following section, but this is not a requirement.

The creation of a meaningful Diagnostic Image is reliant on theselection and definition of meaningful sources of state and their owncontributions to the image. In one embodiment, these sources are madeknown to the Image Manager so that they may readily participate in animage capture.

To qualify as an Image Source a component can implement the ImageSourceinterface and therein must implement a method to stream its diagnosticimage content out to a given output stream. The form and content of thatimage can be left to the specification of each component that wishes todefine itself as an Image Source.

Each Image Source can register itself with the ImageManager by a Stringname which may be used subsequently to unregister the Image Source ifnecessary or to over-register a new instance in place of the currentinstance.

The Image Manager can determine that an Image Source has spent too muchtime creating its image and can send a timeout signal to the source. Thesource is not required to react immediately to the timeout but any workcompleted after the timeout signal may be disregarded. Further the ImageManager can choose to close the OutputStream provided to the ImageSource at any time after a timeout call is made. This allows the ImageManager to gracefully handle Image Sources that may be compromised bythe current failure state or that are exceeding defined performancerequirements.

In one embodiment, it is assumed no Image Source can spend more thanfifteen seconds writing its image content. Any exceptions may be subjectto a timeout from the Image Manager and should be carefully consideredfor their suitability in the Diagnostic Image.

The definition and registration of Image Sources can be left to thecomponent developers themselves. As such this feature's value isdependent on those identified subsystems defining the appropriate imagesources in their subsystem and providing high quality well defined imagecontent.

The presence of well defined Image Sources is only useful if a mechanismexists to request the capture of an image. The Image Manager can providethis support. The capture functionality may be invoked by a firstfailure detection, a configured watch notification, pushbuttonfunctionality exposed in a diagnostic tool, or by a direct API call.

The ImageManager can provide an interface that allows for the capture ofa Diagnostic Image. While creating the image the Image Manager invokeseach Image Source that has been registered. If a particular Image Sourcefails the Image Manager can note this failure in the artifact andcontinue processing the remaining Image Sources. Finally, the ImageManager can add a summary artifact as detailed below.

If the image capture time is deemed too substantial for synchronousinvocation, the capture implementation may choose to immediately returnand manage the capture process asynchronously.

The ImageManager can provide a means of modifying the configureddestination of the output image artifact. This value can be persistedand can survive server restart. The Image Manager can also provide forrequest level overrides of the destination directory with overloadedimageCapture methods. The request level override can supersedes theconfigured destination. If the Image Manager is unable to write to thespecified directory or fails during writing an exception can begenerated.

During initialization the ImageManager can test the configureddestination directory for writability. If the test fails, an errormessage can be logged.

The Image Manager can include in each Diagnostic Image a summaryartifact that can include, but not be limited to the following elements:

-   -   Creation date and time of the image    -   Source of the capture request    -   Name of each Image Source included in the image and the time        spent processing each of those Image Sources    -   JVM and OS information if available    -   Commandline arguments if available    -   Networking muxer version if available    -   WLS version including patch and build number information

The creation of a Diagnostic Image may in fact be a costly exercise andas such it should be an infrequent task. Creating an image for eacherror detected is neither efficient nor likely more effective than asingle image created on the first failure. Considering that errors areoften clustered, a lockout period can be imposed prohibiting the ImageManager from creating a secondary image until after some timeout.

The default lockout can be a predetermined time period such as 15minutes, but the interface can provide an override in which the lockoutmay be set higher or lower. In one embodiment, the lockout value isneither perpetuated nor persisted. Once the lockout time has passed, acapture request may be handled and its subsequent lockout can again be15 minutes unless specified with another lockout override.

Capture requests that occur during a lockout can result in aCaptureLockoutException, which can be safely ignored unless in factthere is proof that no such image has been created, and can bedisregarded. This functionality is necessary to avoid a cascade oferrors generating many similar image artifacts.

If an image capture fails for any reason including I/O problems or afatal error in a registered Image Source the lockout can not be set andthe Image Manager can be immediately ready to handle another capturerequest.

A captured image can result in a single file artifact for the entireserver which is uniquely named. More than one image artifact, withdistinct names, may exist at a given time. In one embodiment, each imageartifact is named as follows:

-   -   diagnostic_image_domain_server_yyyy_MM_dd_HH_mm_ss

If for some reason, and it should be noted this is an extremely unlikelycase (given that image creation should take more than one second) a newimage defines an artifact name that is already in use it can result in alog message indicating the collision and can generate an image of thesame name with an additional sequence number appended as follows:

-   -   diagnostic_image_domain_server_yyyy_MM_dd_HH_mm_ss_#

Where # represents the next available sequential number after allexisting files with the same initial image name.

The Image Manager can log the beginning of the Diagnostic Image captureand the completion of the capture as two distinct events. The completionlog message can include the time elapsed during image creation.

Notice that these messages make it possible, using the Watch andNotification capabilities of the Diagnostic Framework discussed below,to watch for these messages and create a notification, by email forexample and that an image capture has occurred.

The console can expose a means of manually invoking an image captureincluding the option to override the destination of the image artifact.Notice that the ImageManager provides no support for selecting what dataor which image sources might be included in an image and this interfacelikewise need not provide such controls.

The console can also wish to expose a link or download page thatprovides a list of known image artifacts and some metadata like creationdata for a particular server suitable for download to theadministrator's system.

A First Failure (prefailure) is a detection by the server that it istransitioning into a failure state. It may in fact recover from thisstate but this failure is the first of what may represent a chain offailures. As such a First Failure is a candidate to trigger theautomatic creation of a Diagnostic Image.

An illustrative scenario is a server transitioning into a failed statein the middle of the night unmonitored and the NodeManager noting thisstate and restarting the server. The administrator can arrive the nextday to find a restarted server and no artifact to indicate the statebefore the restart. For this reason the transition of a server into afailed state is a good First Failure candidate.

When a critical failure is detected a call to the fail mechanism on theFirstFailure interface can be made. This utility method abstracts thedetails of dispatching the capture call to the Image Manager and mayvery well be an asynchronous call.

A failure can be defined by the state of several metrics or healthstates in the server. The definition of these complex monitored statesis handled by the Watch and Notification capabilities of the diagnosticframework. As such one possible notification handling mechanism might infact be a trigger of the First Failure functionality. This notificationhandler type indicates richer means of defining First Failure detectionand providing administrators with a way to automate image capture forconditions custom to their environment.

The value of the Diagnostic Image is a direct function of the ImageSources selected to contribute to the image. The list of expected ImageSources has not been finalized but can include:

-   -   WorkManager state    -   Configuration state    -   Defined deployment plans    -   Plug in configuration (if available)    -   Log Cache (recently logged messages for all logs)    -   State of all known Harvestable instances and their attributes    -   JNDI dump    -   JRockit JRA output file    -   JVM state (thread dump, memory information, etc.)    -   Native info (like file descriptor counts)

Automated Diagnostic Image creation can be driven by the detection ofFirst Failure (prefailure) states. In addition to the transition of theserver into a “failed” state there may be other candidates for FirstFailure detection.

Non-Limiting Example of Diagnostic Instrumentation

The Diagnostic Instrumentation System (DIS) can provide a flexiblemechanism for selectively adding diagnostic code to server software andapplications running on it. It can allow flexible selection of locationsin the server and application code, at which such code is added. Theprocess of adding diagnostic code can be deferred to the time of runningthe server at a deployment site. It also can allow flexible selection ofdiagnostic actions, which are executed at selected locations. Executionof the diagnostic code can be dynamically enabled or disabled while theserver is running. Also, under certain circumstances, the behavior ofthe diagnostic code executed at such locations can be dynamicallychanged while the server is running.

Visibility into the working of a system is essential to make itsupportable. It is desirable that diagnostic information on the systembe available at runtime to identify problems in a running server. Oneway to provide this visibility is to build diagnostic code into thesystem itself during its design and development phase. However, manuallyscattering diagnostic code in the sources has some problems. Firstly, itis difficult to manage and update. Secondly, exposed visibility islimited to the program locations which were deemed interesting duringdevelopment phase. If a problem surfaces elsewhere during productionphase, the diagnostician has to resort to the cumbersome process ofbuilding and deploying debug patches, etc. This process is cumbersome.Therefore, it makes it difficult to identify and resolve transientproblems in a server, which may be running for a long time.

The Diagnostic Instrumentation System can address this problem byproviding a flexible mechanism to add diagnostic code at a post-buildstage. The locations where diagnostic code can be added are calleddiagnostic joinpoints. The Diagnostic Instrumentation System allowsidentification of such diagnostic joinpoints with an expression in ageneric manner. The set of joinpoints identified by such a genericexpression is called a pointcut. The Diagnostic Instrumentation Systemcan provide a mechanism to allow execution of specific diagnostic codeat such pointcuts. The Diagnostic Instrumentation System adds suchdiagnostic code into the server and application code. Since this is donein a post-build stage, it is possible to tune it at a customer siteaccording to specific needs. The diagnostic code itself can beselectively and dynamically enabled or disabled without restarting theserver. Thus, it can be enabled only when problems need to be lookedinto. During normal operation, the diagnostic code can introduce littleor no overhead.

Another motivation for the Diagnostic Instrumentation System is toprovide a flexible instrumentation service to third party diagnostictools vendors. Often, they resort to performing their own bytecodeinstrumentation, which is difficult to control from the serverperspective. Also, their instrumentation tends to change the shape ofJava classes. Thus, having multiple tools vendors doing their owninstrumentation on the same classes can have unpredictable side effects.By providing the instrumentation service, the tools vendors can beencouraged to stay away from doing their own instrumentation andconcentrate on providing higher level analytical functions.

The goal of the Diagnostic Instrumentation System is to provide latestage dynamism, that is, allow dynamic control over diagnostic code tobe performed as late as the situation permits. It can be available to beused with all JVMs supported by the server software.

In one embodiment, the Diagnostic Instrumentation System does not changethe semantics of the original classes or change program state when thediagnostic code runs. It need not introduce user-visible attributes,methods and interfaces to the classes.

Diagnostic code can be added in a post-build phase, e.g., at thecustomer site. Diagnostic code can be selectively added to specifiedpoints in the server and application code during deployment.

Diagnostic Instrumentation System can provide mechanism to selectivelyand dynamically control execution of diagnostic code, which can beenabled, disabled or in some cases, added or removed without restartingthe server.

The Diagnostic Instrumentation System can provide a library of usefuldiagnostic monitors, which can be deployed at common pointcuts in serverand application code.

The Diagnostic Instrumentation System can provide a library of suitablediagnostic advices which can be used in conjunction with diagnosticmonitors to provide visibility into a running server.

Diagnostic Instrumentation System can make it possible to define customdiagnostic monitors, which can allow executing appropriate diagnosticadvices from the advice library at user selectable points in theapplications.

The Diagnostic Instrumentation System can be usable across differentJava virtual machines. When underlying VM is JRockit, it can allowadding diagnostic code into selected classes without restarting theserver or redeploying applications.

Similar to providing visibility into the server, the DiagnosticInstrumentation System can provide visibility into applications.

The Diagnostic Instrumentation System can provide a mechanism to adddiagnostic code into the classes after the product is built. Thus, itcan be possible to delay the decisions regarding what needs to bediagnosed. For example, it can be possible to specify and tune thediagnostic code that can be added to the system at the customer site.Appropriate diagnostic code can be typically added into the classes whenthey can be loaded into the system.

The Diagnostic Instrumentation System can work with the notion ofDiagnostic Monitors. A Diagnostic monitor can be the unit of diagnosticcode. It can identify the locations in the program where the diagnosticcode can be added, as well as the diagnostic actions which can beexecuted at those points. The Diagnostic Instrumentation System canprovide a library of useful diagnostic monitors. The DiagnosticInstrumentation System can weave the diagnostic monitors into server andapplication classes according to the Diagnostic Instrumentation Systemconfiguration.

Diagnostic monitors can be woven into different classes at differentstages during a server life cycle. For eligible server classes, they canbe woven when those classes are loaded into the VM. These classes neednot be unloaded, so the added diagnostic code can always stay in theseclasses when the server is running. The diagnostic code can be removedfrom these classes by changing the Diagnostic Instrumentation Systemconfiguration and restoring the server. However, the DiagnosticInstrumentation System can allow such code to be enabled or disabled ondemand. It can be possible to change the behavior of certain types ofdiagnostic monitors (Delegating Diagnostic Monitors).

Similar to server classes, diagnostic monitors can be woven intoapplication classes. Diagnostic code can be added to such classes whenthey are loaded into the VM after applications are deployed. TheDiagnostic Instrumentation System can allow such code to be enabled ordisabled on demand; or in some cases, change its behavior. Diagnosticcode in applications can be changed without necessarily restarting theserver. For example, it can be changed by updating DiagnosticInstrumentation System configuration for the application and redeployingthe application. When application classes are reloaded, the newDiagnostic Instrumentation System configuration can take effect.

JRockit VM allows replacement of class implementations in a runningprocess. Therefore, when the server is running on a JRockit VM,diagnostic monitors can be re-woven into classes without restarting theserver or redeploying applications. However, this functionality can beavailable only when underlying VM (e.g. JRockit) provides hot-swapfunctionality.

Although this feature provides functionality for late stage weaving, itshould be noted that the process of weaving is potentially expensive.Therefore, using it pervasively can adversely affect server startuptime, application deployment time, etc. The Diagnostic InstrumentationSystem will not preclude weaving diagnostic monitors at build time. Ifcertain diagnostic monitors should always be in the code, it is betterto weave them in at build time.

The process of weaving diagnostic monitors into server and applicationclasses can be controlled with the instrumentation configuration, whichcan be part of the diagnostic framework configuration information. Theweaving can happen when classes are loaded into the VM by differentclassloaders employed by the server. The Diagnostic InstrumentationSystem can control the weaving process within the scope of suchclassloaders. Weaving into server classes for a Server can be controlledby the instrumentation configuration scoped within that server. Weavinginto application classes can be controlled by instrumentationconfiguration within the deployment descriptors of respectiveapplications.

The instrumentation element can define the scope of instrumentation. Alloptions specified within the element can be applicable only to classeswithin its scope. It has the following attributes:

-   -   include: Optional, comma separated regular expressions to        identify classes within the scope, which will participate in the        weaving process.    -   exclude: Optional, comma separated regular expressions to        identify classes within the scope, which will not participate in        weaving.    -   enable: If false, disable weaving while loading classes within        the scope and disable all diagnostic monitors within the scope.        The instrumentation element associated with a server can dictate        how the weaving will be performed for server classes for that        server. The instrumentation elements in application deployment        descriptors can dictate the weaving process within the scope of        the applications. If the optional attribute “include” is        specified, only classes whose names match given regular        expressions will be affected during the weaving process. If the        optional attribute “exclude” is specified, none of the classes        matching with the given regular expressions will be affected.

Diagnostic monitors are manageable units of diagnostic code which can beapplied to pointcuts. An instrumentation element may contain multiplediagnostic monitor elements, which specify the diagnostic activitywithin the scope. They can be of the following types:

-   -   standard-monitor: These diagnostic monitors can define specific        diagnostic behavior at specific pointcuts.    -   delegating-monitor: These diagnostic monitors can define        specific diagnostic pointcuts, where diagnostic activity will        take place. The actual diagnostic activity can be specified by        advice elements enclosed in them.    -   custom-monitor: These diagnostic monitors can allow definition        of custom diagnostic pointcuts within the scope of applications.        Actual diagnostic activity can be specified by advice elements        enclosed in them.        Attributes of the monitor elements can be:    -   name: Display name of the monitor, used to identify the monitor        for control purpose    -   type: Identifies the type of a standard or delegating monitor    -   pointcut: Identifies the pointcut for a custom monitor    -   enable: Identifies if the monitor is to be dynamically enabled.

Advice elements enclosed within delegating-monitor or custom-monitorelements can specify the diagnostic activity that will take place atpointcuts defined by the respective monitors. Attributes of the adviceelements can be:

-   -   name: Display name of the advice, used for control purpose    -   type: Identifies the type of the advice    -   dye_mask: Controls dye filtering for the advice    -   properties: Stringified advice specific properties

Following is a partial list of error messages that can be recorded inthe log file:

-   -   Malformed instrumentation descriptor elements.    -   Specified monitor cannot be used with identified scope.    -   Specified monitor is not available in monitor library.    -   Specified advices are not available in advice library.    -   Incompatible advices vs. delegating monitors.

For certain diagnostic monitors, which can always be woven in, it wouldmake sense to weave them at build time rather than at class load time.This is especially relevant for server classes. Weaving such monitors atbuild time would eliminate weaving overheads at runtime. An ant task toweave monitors at build time can be provided.

Changing classes can be perceived as an unacceptable security risk bysome customers. The diagnostic instrumentation feature can be disabledby default, unless explicitly enabled by a command line option. Also,instrumentation configuration can be protected such that it can bealtered only by users with an administrator role.

The Diagnostic Instrumentation System can capture weaving statistics andmake it available for other components of WLDF for further processing.Some of the statistics can be:

-   -   Number of classes inspected for weaving    -   Number of classes modified    -   Minimum processing time for a class    -   Maximum processing time for a class    -   Total processing time spent in weaving activity

Although late stage weaving is more flexible than build time weaving, itintroduces weaving overheads, which can result in slower server startuptimes and application deployment times. If certain monitors shouldalways be present, they should be woven in as part of the build process.

In one embodiment, after the diagnostic monitors are woven into a classat class loading time, their structure can not be altered unless theserver is restarted or applications redeployed. However, the DiagnosticInstrumentation System can provide some dynamic control over them at runtime. Such monitors can be dynamically enabled or disabled. In somecases, the behavior of such monitors can also be changed at runtimewithout restarting the server or redeploying an application.

Additionally, if the server is running on JRockit JVM, using its classhot-swap functionality, monitors can be rewoven into classes. It canallow swapping in and out diagnostic monitors on demand at runtime.

The Diagnostic Instrumentation System service level API can be exposedthrough InstrumentationManager service. Access to InstrumentationManagercan be protected with WLS security framework, so that only authorizedusers will be able to use it. Broadly speaking, it can provide thefollowing types of services:

The InstrumentationManager can provide functionality to query whatfeatures of the Diagnostic Instrumentation System are available in aparticular installation. It can provide information on availablemonitors, available advices, configured monitors, etc.

The InstrumentationManager can provide functionality to enable ordisable diagnostic monitors.

The InstrumentationManager can provide functionality to attach or detachan advice from the list of available advices to diagnostic monitors.When no advices are attached to a monitor, it can effectively becomedisabled.

This feature can be available when the underlying JVM is JRockit. It canallow weaving in monitors into specified classes on demand at runtime.

Any changes made at runtime can be persisted to the configuration sothat when the server restarts, the changes can apply automatically.Thus, when monitors are enabled or disabled or when advices are attachedor detached to monitors, the configuration can survive server restart.

The Diagnostic Instrumentation System can collect and publish itsruntime metric information. The Diagnostic Instrumentation System canexpose its runtime control API through the interfaceInstrumentationManager.

The Diagnostic Instrumentation System can provide a library ofdiagnostic monitors, which can perform common diagnostic functions. TheDiagnostic Instrumentation System model is extensible in the sense thatit can be easy to create new diagnostic monitors. However, addingadditional code into server and application classes has securityimplications. The Diagnostic Instrumentation System can restrict itselfto diagnostic monitors supported by the diagnostic framework. As newrequirements come up, new diagnostic monitors can be developed.

Monitor implementations in one embodiment are tied to pointcuts. Inother words, once a diagnostic monitor is created, it implicitlyidentifies the locations in the program which it affects. Some monitorscan be applied to server classes only. Others can be applied toapplications. The Diagnostic Instrumentation System can make adistinction between monitors which apply to server classes and thosewhich apply to application classes.

Different categories of diagnostic monitors are supported by theDiagnostic Instrumentation System. Diagnostic monitors can have twocomponents. The pointcuts used by them identify the locations affectedby them. The business logic, or the advices specify the diagnosticactivity that takes place at those locations. A standard monitor uses afixed pointcut and provides a fixed advice. Thus, both components arespecified by the developer of the standard monitor. Delegating monitorscan use a fixed pointcut. However, their advices can be configurable byan administrator. A custom monitor can allow an administrator to definea pointcut and configure advices to be used with it. Following tablesummarizes the different categories of diagnostic monitors of oneembodiment.

Monitor Category Pointcut Advice Standard Monitor Fixed Fixed DelegatingMonitor Fixed Configurable Custom Monitor Configurable ConfigurableThe Diagnostic Instrumentation System can provide following categoriesof diagnostic monitors.Standard Monitors

The standard monitors in the diagnostic monitor library can performspecific diagnostic functions at specific pointcuts. In other words,they can identify the locations in the program which can be affected bythe weaving process. At the same time, they define the actual diagnosticfunctionality, which can be performed at those locations. Once woven in,they can be dynamically enabled or disabled at runtime. However, theirbehavior does not change. A standard monitor in the library is known byits type. A standard monitor of a particular type can be applied to morethan one scope (e.g., application). However, within a scope, a standardmonitor of a particular type can be applied at most once. At runtime,active standard monitors can be identified by their scopes and displaynames.

When standard monitors are disabled at runtime, they introduce minimalperformance overhead. In order to remove them completely from anapplication scope, it can be necessary to remove them from theconfiguration and redeploy the application. To remove a standard monitorcompletely from server scope, it can be necessary to remove it from theconfiguration and restart the server.

Following is the list of standard monitors in the diagnostic monitorlibrary. Note that, this list is likely to grow as more diagnosticrequirements surface from subsystems.

DyeInjectionMonitor

This diagnostic monitor can be responsible for injecting dyes into thediagnostic context when requests enter the system. It can apply topointcuts which identify code in server classes, which handles incomingrequests. This monitor can be available only for the server scope. Whenenabled, it can set certain dye flags in the Diagnostic Context based onits configuration properties and certain request attributes. In oneembodiment, the Diagnostic Context contains a dye vector, which contains64 dye flags. Dye flags [56 . . . 63] are available for application use.Dye flags [0 . . . 55] are reserved for diagnostic framework use.Following table describes the dye flags which are set when requestcharacteristics satisfy certain criteria in one embodiment.

Dye Index Criterion Condition 0 Client Address matches value of propertyADDR0 1 matches value of property ADDR1 2 matches value of propertyADDR2 3 matches value of property ADDR3 4 User identity matches value ofproperty USER0 5 matches value of property USER1 6 matches value ofproperty USER2 7 matches value of property USER3 8 Value of cookie namedmatches value of property COOKIE0 “weblogic.wldf.dye” (HTTP only) 9matches value of property COOKIE1 10 matches value of property COOKIE211 matches value of property COOKIE3 12 Execute queue name matches valueof property EXECQ0 13 matches value of property EXECQ1 14 matches valueof property EXECQ2 15 matches value of property EXECQ3 16 Thread groupname matches value of property THREADGROUP0 17 matches value of propertyTHREADGROUP1 18 matches value of property THREADGROUP2 19 matches valueof property THREADGROUP3 20 Request protocol T3 21 HTTP 22 RMI 23 SOAP24 IIOP 25 JRMP 25 SSL

If the property is not specified, corresponding dye flag is not set. Thecookie is checked only for HTTP requests. If it does not exist,corresponding dye flag is not set. The dye vector flags can be set onrequest entry only when this monitor is deployed in a server scope andis enabled. It captures certain request characteristics and is carriedthrough the diagnostic context.

The dye vector in the Diagnostic Context can be accessed and used bydiagnostic advices described later in this document. If a dye mask isspecified with the advices, they can execute their diagnostic actiononly if the dye vector in the Diagnostic Context is consistent with thedye mask configured for the advice.

The DyeInjectionMonitor diagnostic monitor can make it possible toselectively perform diagnostic actions only on requests which matchabove criteria. It can keep the volume of generated diagnosticinformation to manageable levels, and also allow diagnosis of selectedrequests without slowing down all other requests in a production system.

HttpSessionDebugMonitor

This monitor can help diagnose problems with HTTP sessions in adevelopment environment. It is not intended for production environment.It can be applied to an application scope. When deployed and enabled, itcan affect following pointcuts in the applications.

-   -   Just after a new session is returned.    -   Just after an attribute is added to the session.    -   Just after an attribute is removed from the session        At affected locations, it can display the size of the session to        stdout. If the session contains non-serializable data, an error        message can be displayed to stdout as well.        Delegating Monitors

The delegating monitors in the diagnostic monitor library are a moreflexible version of the standard monitors. They can affect specificpointcuts. However, they do not define specific diagnostic activity ataffected locations by themselves. Instead, they provide empty shells sothat different diagnostic advices implementing specific functionalitycan be plugged into them. Diagnostic advices are described later in thisdocument.

Delegating monitors can provide an extra level of dynamism. Similar tostandard monitors, they can be enabled or disabled at runtime withoutrestarting the server or redeploying applications. Additionally, theirdiagnostic behavior can be changed by changing the set of diagnosticadvices associated with them. Such change in diagnostic behavior can beachieved without server restart or application redeployment. Moreover,multiple diagnostic advices can be plugged into them at the same time.Divergent diagnostic functions can be executed at the same locations,without affecting each other. When multiple advices are attached to adelegating monitor, they can be executed in the same order in which theyare registered with the monitor.

Implementation of delegating monitors may have certain structuralrequirements. For example, some monitors may affect locations at methodentries. Other may affect method entries as well as exits. Consequently,they may require that they can be used with only certain types ofadvices from the advice library. In other words, there is a notion ofcompatibility among delegating monitors and their associated advices.With a particular delegating monitor, only compatible advices may beused.

Delegating monitors can work on certain standard pointcuts such as J2EEpointcuts for applications. When delegating monitors are disabled atruntime, they can introduce minimal performance overhead. In order toremove them completely from an application scope, they can be removedfrom the configuration and the application redeployed. To remove adelegating monitor completely from server scope, it can be necessary toremove it from the configuration and restart the server.

Following is a list of delegating monitors in the diagnostic monitorlibrary. Note that, this list is likely to grow as more diagnosticrequirements surface from subsystems. Although delegating monitorsdefined in the following tables are scoped to applications, serverscoped delegated monitors are not precluded.

Following table lists exemplary monitors that can execute diagnosticadvices before the joinpoints covered by their pointcuts are reached.When enabled, they can execute all attached advices. Only advices ofcompatible types may be attached with the monitors, as indicated in thetable.

Compatible Advice Monitor Name Scope Type PointcutsServlet_Before_Service Application Stateless At method entries ofservlet/jsp Action methods: HttpJspPage._jspService Servlet.serviceHttpServlet.doGet HttpServlet.doPost Filter.doFilterServlet_Before_Session Application Stateless Before calls to servletmethods: Action HttpServletRequest.getSessionHttpSession.setAttribute/putValue HttpSession.getAttribute/getValueHttpSession.removeAttribute/remove Value HttpSession.invalidateServlet_Before_Tags Application Stateless Before calls to jsp methods:Action Tag.doStartTag Tag.doEndTag JNDI_Before_Lookup ApplicationStateless Before calls to Action javax.naming.Context lookup methodsContext.lookup* JMS_Before_TopicPublished Application Stateless Beforecall to methods: Action TopicPublisher.publish JMS_Before_MessageSentApplication Stateless Before call to methods: Action QueSender sendJMS_Before_AsyncMessageReceived Application Stateless At entry ofmethods: Action MessageListener.onMessage JMS_Before_SyncMessageReceivedApplication Stateless Before calls to methods: ActionMessageConsumer.receive* JDBC_Before_GetConnection Application StatelessBefore calls to methods: Action Driver.connect DataSource.getConnectionJDBC_Before_CloseConnection Application Stateless Before calls tomethods: Action Connection.close JDBC_Before_CommitRollback ApplicationStateless Before calls to methods: Action Connection.commitConnection.rollback JDBC_Before_Statement Application Stateless Beforecalls to methods: Action Connection.prepareStatementConnection.prepareCall Statement.addBatch RowSet.setCommandJDBC_Before_Execute Application Stateless Before calls to methods:Action Statement.execute* PreparedStatement.execute*EJB_Before_SessionEjbMethods Application Stateless At entry of methods:Action SessionBean.setSessionContext SessionBean.ejbRemoveSessionBean.ejbActivate SessionBean.ejbPassivateEJB_Before_SessionEjbSemanticMethods Application Stateless At entry ofmethods: Action SessionBean.ejbCreate SessionBean.ejbPostCreateEJB_Before_SessionEjbBusinessMethods Application Stateless At entry ofall SessionBean Action methods, which are not standard ejb methods.EJB_Before_EntityEjbMethods Application Stateless At entry of methods:Action EnitityBean.setEntityContext EnitityBean.unsetEntityContextEnitityBean.ejbRemove EnitityBean.ejbActivate EnitityBean.ejbPassivateEnitityBean.ejbLoad EnitityBean.ejbStoreEJB_Before_EntityEjbSemanticMethods Application Stateless At entry ofmethods: Action EnitityBean.set* EnitityBean.get* EnitityBean.ejbFind*EnitityBean.ejbHome* EnitityBean.ejbSelect* EnitityBean.ejbCreate*EnitityBean.ejbPostCreate* EJB_Before_EntityEjbBusinessMethodsApplication Stateless At entry of all EnitityBean Action methods, whichare not standard ejb methods. MDB_Before_MessageReceived ApplicationStateless At entry of methods: Action MessageDrivenBean.onMessageMDB_Before_SetMessageDrivenContext Application Stateless At entry ofmethods: Action MessageDrivenBean.setMessageDriven ContextMDB_Before_Remove Application Stateless At entry of methods: ActionMessageDrivenBean.ejbRemove TX_Before_Start Application Stateless Atentry of methods: Advice UserTransaction.begin TX_Before_CommitApplication Stateless At entry of methods: Advice UserTransaction.commitTX_Before_Rollback Application Stateless At entry of methods: AdviceUserTransaction.rollback BeforeJCAMonitor Application Stateless Beforecalls to JCA methods: Advice ConnectionFactory.getConnectionConnection.close

Following table lists exemplary monitors that can execute diagnosticadvices after the joinpoints covered by their pointcuts. When enabled,they can execute all attached advices. Only advices of compatible typesmay be attached with the monitors, as indicated in the table.

Compatible Advice Monitor Name Scope Type PointcutsAfterEntityBeanMethodsMonitor Application StatelessAdvice At exit ofentity bean methods: setEntityContext unsetEntityContext ejbRemoveejbActivate ejbPassivate ejbLoad ejbStoreAfterEntitySemanticMethodsMonitor Application StatelessAdvice At exit ofentity bean methods: setters getters findByPrimaryKey ejbFind methodsejbHome methods ejbSelect methods ejbCreate methods ejbPostCreateAfterEntityBusinessMethodsMonitor Application StatelessAdvice At exit ofall public entity bean methods not covered by BeforeEntityBeanMethodsand BeforeEntitySemanticMethods. AfterJCAMonitor ApplicationStatelessAdvice After calls to JCA methods:ConnectionFactory.getConnection Connection.closeAfterJDBCConnectionMonitor Application StatelessAdvice After calls toJDBC methods: Driver.connect DataSource.getConnection Connection.commitAfterJDBCExecuteMonitor Application StatelessAdvice After calls to JDBCmethods: Statement.execute* PreparedStatement.execute*AfterJDBCStatementMonitor Application StatelessAdvice After calls toJDBC methods: StringArgumentAdvice Connection.prepareStatementConnection.prepareCall Statement.addBatch AfterJMSMonitor ApplicationStatelessAdvice After calls to JMS methods: TopicPublisher.publishQueueSender.send MessageConsumer.receive* At entry of following methods:MessageListener.onMessage AfterMDBMonitor Application StatelessAdvice Atmethod exits of MDB methods: MessageDrivenBean.setMessageDrivenContextMessageDrivenBean.ejbRemove AfterServletServicesMonitor ApplicationStatelessAdvice At method exits of servlet/jsp methods:HttpJspPage._jspService Servlet.service Filter.doFilterAfterServletSessionMonitor Application StatelessAdvice After calls toservlet methods: HttpServletRequest.getSession HttpSession.setAttributeHttpSession.getAttribute HttpSession.removeAttributeHttpSession.removeValue HttpSession.invalidate AfterServletTagsMonitorApplication StatelessAdvice After calls to jsp methods: Tag.doStartTagTag.doEndTag AfterTxMonitor Application StatelessAdvice After calls toUserTransaction methods: begin commit rollback

Following table lists exemplary monitors that can execute diagnosticadvices around the joinpoints covered by their pointcuts. When enabled,they can execute all attached advices. Only advices of compatible typesmay be attached with the monitors, as indicated in the table.

Compatible Monitor Name Scope Advice Type PointcutsAroundEntityBeanMethodsMonitor Application AroundAdvice At entry andexit of entity bean methods: setEntityContext unsetEntityContextejbRemove ejbActivate ejbPassivate ejbLoad ejbStoreAroundEntitySemanticMethodsMonitor Application AroundAdvice At entry andexit of entity bean methods: setters getters findByPrimaryKey ejbFindmethods ejbHome methods ejbSelect methods ejbCreate methodsejbPostCreate AroundEntityBusinessMethodsMonitor ApplicationAroundAdvice At entry and exit of all public entity bean methods notcovered by BeforeEntityBeanMethods and BeforeEntitySemanticMethods.AroundJCAMonitor Application AroundAdvice Before and after calls to JCAmethods: ConnectionFactory.getConnection Connection.closeAroundJDBCConnectionMonitor Application AroundAdvice Before and aftercalls to JDBC methods: Driver.connect DataSource.getConnectionConnection.commit AroundJDBCExecuteMonitor Application AroundAdviceBefore and after calls to JDBC methods: Statement.execute*PreparedStatement.execute* AroundJDBCStatementMonitor ApplicationAroundAdvice Before and after calls to JDBC methods:Connection.prepareStatement Connection.prepareCall Statement.addBatchAroundJMSMonitor Application AroundAdvice Before and after calls to JMSmethods: TopicPublisher.publish QueueSender.sendMessageConsumer.receive* At entry of following methods:MessageListener.onMessage AroundMDBMonitor Application AroundAdvice Atmethod entry and exits of MDB methods:MessageDrivenBean.setMessageDrivenContext MessageDrivenBean.ejbRemoveAroundServletServicesMonitor Application AroundAdvice At method entryand exits of servlet/jsp methods: HttpJspPage._jspServiceServlet.service Filter.doFilter AroundServletSessionMonitor ApplicationAroundAdvice Before and after calls to servlet methods:HttpServletRequest.getSession HttpSession.setAttributeHttpSession.getAttribute HttpSession.removeAttributeHttpSession.removeValue HttpSession.invalidate AroundServletTagsMonitorApplication AroundAdvice Before and after calls to jsp methods:Tag.doStartTag Tag.doEndTag AroundTxMonitor Application AroundAdviceBefore and after calls to UserTransaction methods: begin commit rollbackCustom Monitors

The standard and delegating monitors can work on specific standardpointcuts identified during their development. They may satisfy most ofthe diagnostic requirements. However, there can be situations whendiagnostic code needs to be added at specific methods in applicationclasses. Custom monitors provide the means to execute diagnostic advicesat locations in application code. These locations can be identifiedwhile configuring custom monitors by specifying appropriate pointcuts.As part of defining a custom monitor, following information can berequired:

-   -   name: Display name of the custom monitor, which can be used for        control purposes.    -   scope: Identify the application scope to which the monitor can        apply.    -   pointcut: An expression, which identifies the set of locations        in the applications which can be affected.    -   compatibility: Type of compatible advices    -   advice: The diagnostic advice that can be executed at the        pointcut.        The pointcut syntax can identify a method execution location, or        a method callsite location.

Custom monitors can be enabled or disabled at runtime. When disabled,they can introduce minimal performance overhead. In order to remove themcompletely, they can be removed from the configuration and theapplication redeployed. Similarly, a newly configured or updated custommonitor can take effect only after the application is redeployed.

A custom monitor can accept only advices of compatible types. Dependingon the advice, it can be executed before, after or around the joinpointsidentified by the pointcut. Custom monitors can be available only forapplication scopes. That is, in one embodiment, the DiagnosticInstrumentation System does not allow custom monitors to be used withserver classes in production mode. However, custom monitors may be madeavailable to server developers to be used on server code in developmentmode.

As described above, the delegating and custom monitors only identify thelocations where diagnostic activity takes place. However, they do notprovide the diagnostic functionality of their own. The diagnosticfunctionality is provided by the diagnostic advices, which can beattached to the delegating or custom monitors. the DiagnosticInstrumentation System can provides a library of diagnostic adviceswhich can be used with the delegating monitors provided in the monitorlibrary.

Depending on the functionality of an advice, it may need certainenvironment to do its job. Such environment is provided by thedelegating monitor to which it is attached. Therefore, diagnosticadvices can be used only with compatible monitors. Diagnostic advicesare classified by their types. Such types are used to identifycompatibilities by the diagnostic instrumentation system.

The Diagnostic Instrumentation System can allow using only advices onlyfrom the advice library and not any arbitrary code. However, diagnosticframework developers can easily add new advices as new requirementssurface. For security reasons, dynamic addition to advice and monitorlibraries need not be supported.

In one embodiment, the diagnostic activity of an advice can be executedonly if certain filtering conditions are met, based on the dye vector inthe diagnostic context. The dye vector can be typically set by theDyeInjectionMonitor diagnostic monitor at the request entry. The dyevector carries request characteristics as the request progresses throughits execution path.

If dye filtering for the advice is disabled, the business logic of theadvice is unconditionally executed.

Otherwise, the current dye vector in the diagnostic context must beconsistent with the dye mask configured with the advice. In oneembodiment, the business logic of the advice can be executed only if thedye vector in the Diagnostic Context satisfies the following condition:

-   -   (dye_mask & dye_vector=dye_mask)

By properly configuring and deploying the DyeInjectionMonitor andconfiguring the dye mask of the advice, the diagnostic advices can beexecuted only for requests of special interests. For example, a testrequest for a test machine can be fired and its progress can beanalyzed. This can facilitate quick analysis of diagnostic data. It canalso ensure that other requests are not slowed down by diagnosticactivity.

Configuring the dye masks of individual advices can be a tedious task.To simplify matters, the InstrumentationManager can provide helpermethods to set the dye masks in generic manner.

The Diagnostic Instrumentation System advice library can containfollowing exemplary diagnostic advices.

Trace2LogAdvice

This advice can cause a trace log entry to be made in the server log.Its type is StatelessAdvice. The advice implementation can interfacewith the logging system. The log entry can contain:

-   -   TRACE tag    -   [Trace2Log] advice-name    -   Context identifier from the Diagnostic Context which uniquely        identifies the request    -   Thread id    -   User identity    -   Timestamp    -   Identification of location from where the advice got executed        InvocationCounterAdvice

The type of this advice is StatelessAdvice. This advice computes theinvocation counts of associated joinpoints. As the program executes, itcan gather invocation count information in its repository. Collecteddata in its repository can be exposed to the diagnostic frameworkharvester using the HarvestableDataSource interface. Thus, collectedinformation can be further processed.

DisplayArgumentsAdvice

The type of this advice is StatelessAdvice. When executed, this advicecan create entries in the server log file, which can display argumentspassed into the joinpoint, if any.

ElapsedTimeAdvice

The type of this advice is AroundAdvice. When executed, it captures thetimestamp before and after the execution of associated joinpoint. Itthen computes the elapsed time by computing the difference. Its runtimeregistry can gather certain statistical information such as totalelapsed time, average elapsed time, invocation count. Collectedinformation is exposed to the harvester using the HarvestabledataSource.Thus, collected information can be further processed.TraceElapsedTimeAdvice

The type of this advice is AroundAdvice. When executed, this advicecaptures the timestamps before and after the execution of associatedjoinpoint. It then computes the elapsed time by computing thedifference. It can create a log entry in server log with followinginformation:

-   -   TRACE tag    -   [TraceElapsedTime] advice-name    -   Context identifier from the Diagnostic Context which uniquely        identifies the request    -   Thread id    -   User identity    -   Timestamp at entry    -   Elapsed time in milliseconds    -   Identification of location from where the advice got executed        StackDumpAdvice

The type of this advice is StatelessAdvice. When executed, by default,this advice can print the stack dump on stdout. If the FIRST_CALL_ONLYproperty is set to true, the advice can dump stack trace only on thefirst call to the joinpoint.

MemoryStatisticsAdvice

The type of this advice is StatelessAdvice. When executed, this advicecan display information about current, total and maximum memoryavailable to the JVM to stdout.

Most of the functionality provided by the Diagnostic InstrumentationSystem can be portable to all popular JVMs. It can do so by addingdiagnostic code during class load time, which can be implemented in aJVM independent manner. Most JVMs do not support changing classes afterthey have been loaded. Therefore, with most JVMs, changes to classesafter class-loading are possible only if the applications areredeployed.

JRockit VM can provide hot-swap functionality. This can allow theDiagnostic Instrumentation System to provide optimizations to changediagnostic code into selected classes even after they are loaded. Thisfunctionality can be available only for classes within the applicationscopes.

This functionality can allow weaving specific diagnostic monitors into aspecific application class within application scopes. When thisfunctionality is available, the InstrumentationManager described abovecan allow re-weaving and hot-swapping the class without a server restartor redeployment. For example, a custom monitor can be configured; theclass can be re-woven and hot-swapped using the functionality in theInstrumentationManager while the server is running.

Similar to diagnostic support for server code, the DiagnosticInstrumentation System can provide diagnostic support for applications.It can be controlled by defining instrumentation scopes which are scopedto the application. Diagnostic monitors specified in the instrumentationscope can be woven into the application classes when they are loaded. Itcan be possible to update such diagnostic code by changing theinstrumentation configuration and redeploying the application.

Non-Limiting Example of Debug

This document defines an exemplary framework to be built to providesupport for better debugability for the diagnosis of problems orsymptoms occurring in a live server. The information in the server logsis sometimes not sufficient to provide insight into the cause of theproblem. Certain debug settings can be applied, capture the output andonce the problem cause is determined disable the Debug when normaloperations are resumed. These actions can be executed without incurringany system downtime or significant degradation in performance.

Debugging is a low tech, but simple and effective solution in lot ofcases to provide better input to the Support, Engineering folks as wellas Server System Administrator. It helps in figuring out what might behappening in a situation where traditional debugging through theIntegrated Development Environment (IDE) is not available, as in aproduction deployment. Typical debug output can dump additional stateinformation, not included in the normal server logs. The state couldinclude the execution point in the program flow, the values of theparameters being passed or the interim state of objects in atransaction. These snapshots help to reconstruct the scenario causingthe problem and can lead to quicker solutions.

Debugging capability can refer here to the capability to emit some typeof instrumented output that can include details about the current state,so that further analysis can be done to figure out the actual problem.

A ServerDebugMBean can be used as a standard interface for definingdebugging options that are configurable and persisted across serversessions. Client side debug configuration cam be defined using theKernelDebugMbean interface. We can also define a grouping scheme so thatServerDebugMBean attributes can be easily navigated and selected,because the number of attributes is expected to grow dramatically asthis approach is used throughout the server software. In addition debugoutput can be controllable depending on the presence of the DiagnosticContext.

The Debug Provider API can define a lightweight interface that can beused for debugging throughout the server software codebase. This canguarantee that the debug output is formatted consistently and ends up ina standard location.

The ServerDebugMBean and KernelDebugMBean attributes can be used todefine debug flags that enable or disable a specific type of debugoutput. All such debug attributes can be declared as dynamic using the@dynamic metadata tag. This can be the standard approach for introducingdebug code in the server product. Other methodologies that exist todaycan be replaced in favor of the uniform approach being proposed here.When defining debugging code in the code base, careful thought must begiven to ascertain whether this debugging is useful for diagnosingfuture problems. The question to be asked is whether this debug would behelpful to the Support Engineers, call center employees and server usersin determining the cause of a problem they are facing and exposedappropriately using the mechanisms defined here. When the diagnosticcontext is available, it can be used to control the debug output forrequests meeting a certain criteria based on User identity, Protocol,Client IP Address etc.

Other debugging code using DebugCategory, command line switches, javasystem properties etc can be migrated to using the ServerDebugMBeanattributes. This can lead to an attribute explosion on theServerDebugMBean which the users can find it difficult to navigatethrough. A grouping scheme can help users drill down to the desireddebug control easily without being required to have the knowledge of aspecific flag in the context of a particular problem. In order toaddress this issue, the debug flag needs to be arranged in a tree likestructure. To introduce a terminology we can call this a tree of“DebugScope” nodes. Each DebugScope node can contain other DebugScopenodes and ServerDebugMBean attributes thus forming a tree likestructure. The leaves of this DebugScope tree would be theServerDebugMBean attributes. Each DebugScope node can be identified by afully qualified name with the period delimiter being used to specify thenode name of successive children down the tree path.

Since client side debug flags are probably not as numerous as comparedto the server side flags, the grouping scheme need not apply to theKernelDebugMBean interface. Nevertheless all client side debug flags canbe defined as an attribute on the KernelDebugMBean, so that Support hasaccess to all the available debug settings in a single place.

A DebugScope specifies the aspects of a particular kind of debuggingoutput. A DebugScope defines a hierarchical namespace for the debugparameters. In addition a scope can contain other child scopes. EachDebugScope node can contain other DebugScope nodes and ServerDebugMBeanattributes as leaf nodes. A ServerDebugMBean attribute can be mapped tomore than one node in the debug tree. For example a ServerDebugMBeanattribute to debug servlets can be mapped to the DebugScope nodesweblogic.applications.webapp as well as weblogic.protocol.http. TheDebugScope tree definition can be static in nature, defineddeclaratively using javadoc metadata tags in the ServerDebugMBean.javasource file and created during the build process. This can be availablein a read-only mode when the server is running. To draw an analogy tothe file system on a computer disk, the DebugScope nodes are equivalentto directories which can contain child Debug Scopes just likesub-directories. ServerDebugMBean attributes are equivalent to files. Asingle attribute can be placed under more than one DebugScope node justlike a symlink.

To illustrate the DebugScope “weblogic.ejb” is parent to“weblogic.ejb.cache”, “weblogic.ejb.security” and “weblogic.ejb.locking”This would enable users to enable/disable debugging at any desired levelof the hierarchy. If they enable debugging for weblogic.ejb all thechild scopes would be enabled for example cache, security and locking.Or if the users choose to they can just enable/disable the specific onesthey are interested in to a finer level of granularity. Thishierarchical arrangement of the debug settings in a naming scheme meansthat the users need not know upfront whether the problem is in caching,security or locking. They can control the settings at a higher level,but on the other hand if they have some insight into a narrow area wherethe problem may lie, a finer scope may be selected. Also the users donot have to be confronted with umpteen attributes, instead they cannavigate through a tree control based visual representation which theWebLogic Console can provide.

DebugScope is higher levels construct that maps to lower level ways ofcontrolling debugging like ServerDebugMBean attributes. When aparticular DebugScope is selected it can select its entire child DebugScopes and the associated ServerDebugMBean attributes down the tree. Bycreating a higher level construct over the existing ServerDebugMBeanattributes we can be able to present the users with a single view intoour debugging mechanisms. Users can have a coarse grained view of theDebug Scopes which they can drill down into if they want.

The Diagnostic Context can be used to further control the debug output.Users can have the additional capability of specifying that debuggingoutput be emitted only for dyed requests. For example users may haveconfigured DebugEJBCache to be enabled and they want the debug to emitonly for requests that match a specific user identity. This way thevolume of debugging information generated is low and can be easilyconsumed for diagnosis purposes. Context based debugging is not enabledby default; debug can be emitted for all requests. The context baseddebugging provides a throttling mechanism to limit the volume of debuginformation in the log files.

Here is an illustration of how we can define the metadata for theServerDebugMBean attributes to declare where they show up in the tree ofDebug Scopes. The DebugScope node to which an attribute belong can bespecified by the @debugScope javadoc metadata tag. Multiple @debugScopetags can be defined for an attribute, so that it can show up under morethan one nodes in the tree. The interface can be processed to create astatic debugscope tree during the build phase. The documentation for thedebugscopes can be defined at the class level using @debugScopeDoc tags.The interface also includes a configurable switch that enables ordisables context based debugging and a mask value that defines thecriteria when debug is enabled for dyed requests.

The configuration of debug settings can be persisted in the same way asall other MBean attributes. A service level ServerDebugService API canbe provided to navigate the DebugScope tree and support for enabling ordisabling at the DebugScope node as well as the attribute level. Aninstance of this interface can be retrieved by invoking theDiagnosticService interface for a given server.

The debugging output can be in the same location as the server log, thismakes it easier for them to consume this information. The debug messagescan be formatted in a standard way and includes other relatedinformation like the current thread id, user id, transaction id and thecontext correlation id. The debug output can be throttled by anadministrator so that it is emitted only for requests meeting aspecified criterion.

The Debug Provider API defined by the DebugLogger interface can be usedby server Subsystems throughout the code base to emit debug output forboth client and server sides. Users can get a reference to theDebugLogger by name that can appear in the server logs as the subsystemname. This can help segregate the debug messages coming from a specificnamed instance of a DebugLogger. The DebugLogger can provide a finalmethod to check the state of the debug flag which can use cached valuesupdated through JMX AttributeChangeNotifications. The check can alsoevaluate the Diagnostic Context when context based debugging is enabled.

The DebugLogger API can provide a factory method for getting an instanceby name and methods to emit debug output to the log. In addition, it canprovide methods to check whether corresponding the debug flag is enable.This check can evaluate the value of the MBean attribute and theDiagnostic Context when available.

The root level node for the DebugScope can be statically defined to be aname such as “weblogic”. Subsystems can define their own nodesreflecting the subsystem organization that exists for example we couldhave nodes like “weblogic.ejb”, “weblogicjms” OR weblogic.management forEJB, JMS and OA&M respectively. Under each subsystem additional nodescan be defined using functionality rather than strict package layout inthe source tree.

Debug Check Pseudo-code

// The debug check inside DebugLogger.isDebugEnabled( ) can look like //this. In the normal case when a debug flag is disabled it can evaluate// only one boolean value and can be efficient. When debug is enabled //though it can additionally check another boolean value for the context.// Only when the context based debug is ON we go to the Context toretrieve the dye mask and check against the criteria to limit the debugif (debugEnabled && (!contextDebugON || (debugMask & Context.getMask( )== debugMask))) {  // Emit debug code }Debug Log File Message Formatcan includes the following fields:

-   -   Timestamp    -   Severity (in this case Debug)    -   Debug attribute name (DebugMBeanHome in the example below)    -   Machine name    -   Server name    -   Thread name    -   User ID    -   Transaction ID    -   Msg id (‘000000’ for messages not I18ned)    -   Message text    -   Stack trace (optional)    -   Context Id (optional)        ####< Jul. 29, 2003 1:41:23 PM EDT> <Debug> <DebugMBeanHome>        <dwarkamai> <myserver> <main> <<WLS Kernel>> < > <000000>        <Registering configMbean:mydomain:Location=myserver, Name=uddi,        Type=ApplicationConfig for admin changes with adminMbean:        mydomain:Name=uddi,Type=Application>        Non Limiting Example of Watches and Notification

The Watches & Notifications (W&N) system can serve as a primarycomponent of the Diagnostic Framework that enhances the existing JMXmonitoring and notification capabilities. The W&N system can be capableof allowing system administrators to express rules that specifyrelationships among multiple attributes of stored data, such as frommultiple MBeans, and the ability to fire notifications once thespecified rules are triggered. Apart from introducing an enhancedmonitoring facility, the Diplomatic Framework can also introduces anenhanced notification system that allows notifications to be firedthrough disparate transportation mediums, such as SNMP traps, JMS, JMX,and SMTP. The W&N system introduces a notification architecture thatallows the server software to add new transportation mediums easily.System administrators can have the ability to express parameterizedrules for watches, as well as configure various notificationtransportation mediums, via an intuitive graphical interface that can bepart of the administrative console.

The JMX 1.0 specification that provides a primitive monitoring mechanismand notification transport system. The JMX specification essentiallydefines three monitors, namely the StringMonitor, CounterMonitor, andGaugeMonitor. While these monitors are useful for monitoring a singleJMX MBean, they quickly become insufficient for cases where a richer setof dependencies and relationships exist among MBean attributes thatdetermine if a notification should be fired or not. Using aparameterized rule-based feature to express how attributes of anarbitrary number of MBeans should be monitored allows one to easilyexpress monitoring criteria in a concise form. The JMX Notificationmechanism that WLS currently implements only supports sending JMXnotifications to other JMX agents and clients through a standardjavax.management.Notification instance. An enhanced notification systemcan support other transport mechanisms apart from JMX, such as SMTP,SNMP, JMS, and user-specific mediums such as loggers, etc. Once themonitoring and notification transport facilities have been enhanced,system administrators can possess the capabilities to more intelligentlymonitor server software instances and managed resources deployed onthem.

A parameterized rule-based feature becomes a very significant componentof an intelligent monitoring system. The JMX specification describes aquery language that can be used for selecting and filtering MBeansregistered with the MBean Server. This query language is a JMX standardand provides support for various relational, string, numerical, andboolean expressions. However, the manner in which queries areconstructed is a bit cumbersome to utilize, and does not provide auser-friendly expression language that system administrators can quicklyharness to create complex watches that follow certain rules.Furthermore, the JMX MBean Server query language does not inherentlysupport the construction of rules for dealing with watch/monitoring ofvarious MBean attributes, but rather a mechanism for only querying theunderlying MBean Server for MBeans that match certain criteria specifiedin the queries. Therefore, a parameterized rule-based feature can beimplemented, which could provide a user-friendly interface that systemadministrators can utilize to construct watch/monitoring rules thatenhanced monitoring entities could examine and execute. The rules canutilize the query expression language used in the Accessor component ofthe diagnostic framework.

In addition, a watch can have the concept of an alarm. If a watchevaluates to true and is configured as an alarm, then the watch need notbe evaluated again till the alarm is reset. An alarm can either resetautomatically when the specified amount of time has expired or manuallyby an administrator. Alarms are in-memory and can be reset upon serverreboot.

JMX Notifications allow notifications to be transmitted across local andremote JVMs that house an MBean Server running within a server softwareinstance. For applications which adhere to the JMX specification, thistype of notification transport is very feasible and sufficient. However,there are many situations in which it would be advantageous to transportnotifications across other mediums, such as SNMP, SMTP, and JMS. In adistributed environment in which multiple server software instances arepresent, it's more appropriate to transport notifications across thewire through an existing protocol that the underlying receiver/sendercan more easily harness. The server software can provide customnotification listeners that can propagate notifications via disparatemediums such as SMTP, SNMP, and JMS.

The Watch can capture a system administrator's specific watchconfiguration. The system administrator can specify the rule expressionfor the watch by graphically selecting the attributes and relationalexpressions that should be applied before a notification is triggered.System administrators can create multiple “watches” for variousattributes and events (log events, for example) in the system. A Watchartifact encapsulates a Watch rule expression, provides a mechanism toadd/remove notification listeners for a particular watch, and can beenabled or disabled.

The Watch rule expression can uses the same syntax as the Accessorcomponent, but have different variables for different types of ruleexpressions. The expression allows you to specify attributes, operators,and values. In one embodiment, there are three types of ruleexpressions: rules that refer to harvester data attributes, rules thatrefer to log events, and rules that refer to instrumentation events. Acustomer can specify the type of rule expression when creating thewatch, and the watch rule can only refer to variables for that type ofrule expression. Therefore, a rule can not refer to both harvester dataattributes and log event data.

A primary abstraction in the W&N framework is the WatchManager, whichactually manages the Watch artifacts and underlying details ofevaluating rules using the various log events and Harvester. TheWatchManager performs the evaluation of watches, manages the alarms forwatches, and enables/disables specific Watches.

A Watch can encapsulate all of the information for a watch rule. Thisincludes the watch rule expression, the alarm settings for the watch,and the various notification handlers that can be fired once a Watchrule expression evaluates to true.

Note that if an MBean suddenly unregisters from the MBean server, but isstill referenced within a Watch, the particular Watch can be essentiallydeactivated. Additionally, notifications fired on behalf of a particularWatch are not guaranteed to arrive at their final destinations in thecase of a server or subsystem failure.

In one embodiment, a concrete Watch rule expression implementationencapsulates all information that is necessary for specifying aparticular rule and may include any of the following:

-   1. Variable name. This differs based upon the type of rule    expression.-   2. Specified threshold value (String, Integer, Long, Double, Boolean    constant)-   3. Operator (<, >,<=, >=, =, !=, AND, OR, NOT, MATCHES, LIKE)

The syntax for a watch rule expression can be the shared with theAccessor component. The proposed syntax is specified in theweblogic/diagnostics/query/Query.g andweblogic/diagnostics/query/TreeParser.g files. Each type of ruleexpression supports different variables. Variable names for thedifferent types are listed below.

A log event rule expression refers to attributes of a log message fromthe server log. There are rule expression variables for the attributesof the weblogic.logging.LogEntry class. The variable names can be:

Data Variable Description Type Id Id of Log Message. Usually starts with‘BEA-’ String MachineName Name of machine that generated the log Stringmessage. ServerName Name of server that generated the log message.String ThreadName Name of thread that generated the log message. StringUserId Id of User that generated the log message. String TransactionIdJTA Transaction id of thread that generated String the log message.Severity I18N severity text of log message. See Stringweblogic.logging.SeverityI18N for values. Subsystem Name of subsystememitting the log message. String Timestamp Timestamp when the logmessage was created. Long LogMessage Message content of the log message.String Throwable Exception associated with this log event. String Thisis created by calling the toString method of the Exception.

Example Log Event Rule Expression

((Severity=‘Warning’) AND (Id=‘BEA-320012’)

A harvester rule expression refers to attributes that are beingharvested. The customer must specify the complete MBean object name asfollows:

-   ${DomainName:Name=MBeanName,Location=ServerName, Type=MBean    Type,Parent=ParentName,Attribute=AttributeName}

Example Harvester Rule Expression

-   ${mydomain:Name=HarvesterRuntime,Location=myserver,    Type=HarvesterRuntime,ServerRuntime=myserver,Attribute=TotalSamplingCycles}>10    Instrumentation Event Data Variable Names:

An instrumentation event data rule expression can refer to attributes ofa data record from the instrumentation event. There are rule expressionvariables for the attributes of the data record. The variable names are:

Data Variable Description Type Timestamp Timestamp when theinstrumentation event Long was created. ContextId Diagnostic Context Idof instrumentation String event. TransactionId JTA Transaction id ofthread that created String the instrumentation event. UserId Id of Userthat created the instrumentation String event. ActionType Type ofinstrumentation action. String DomainName Name of domain StringServerName Name of server that created the String instrumentation event.ScopeName Name of instrumentation scope String MonitorType Type ofmonitor String SourceFile Source file name String LineNumber Line numberin source file Integer ClassName Class name of join point StringMethodName Method name of join point. String MethodDescriptor Methodarguments of join point String ReturnValue Return value of join point.String Payload Payload of instrumentation event data. String

Example Instrumentation Event Data Rule Expression

(Action Type=‘ThreadDumpAction’)

The WatchManager can be a singleton object whose primary purpose is tomanage the W&N system. The WatchManager performs the following tasks:

-   1. Maintains a list of all watches. Maintains the state of a watch    (enabled, disabled, alarm).-   2. Is called by the Harvester after sampling is complete. Evaluates    watches that refer to harvester attributes and calls back into the    harvester to access harvested data from the current sample.-   4. Is called by the Watch Log handler to evaluate watches that refer    to log event attributes. Log events are queued are evaluated    asynchronously.-   5. Maintains alarms and resets alarms either automatically or    manually.    The WatchManager is not a public class and does not need to be    documented. The public interfaces are the configuration descriptors    and the runtime MBean.

The Watch and Notification runtime MBean can contain methods to returnstatistics about the watch and notification processing performed by theWatch Manager. The MBean can contain methods to return the followinginformation:

-   1. For each type of watch (harvester, log, instrumentation event),    returns statistics about number of evaluation cycles, total number    of watches evaluated, total evaluations to true, and average time    per evaluation cycle.-   2. Current active alarm count and maximum active alarm count.-   3. Number of manual reset alarms and automatic reset alarms that    have occurred.-   4. Current active alarms.    The MBean also contains a method to reset an active alarm.

The administrative console utility can be the primary interaction pointwith the W&N system. The configuration and creation of the Watch andWatch rule expressions can occur when the system administratorgraphically specifies watches and rules for MBean attributes via theadministrative console. Watch rule expressions can also be modified atthe command line via the weblogic.Admin utility or via the WebLogicScripting Tool.

An enhanced notification system can support disparate notificationtransportation mediums to be added to diagnostic framework watchesdynamically. Currently, the JMX notification mechanism only supports JMXas the primary transportation route for notifications. The enhancednotification system of the diagnostic framework can provides a mechanismfor other mediums such as SNMP, SMTP, and JMS to be utilized whennotifications are triggered. The Watches feature described in theprevious section can utilize the enhanced notification system todynamically emit notifications when Watch rules are satisfied.

Watch Notifications can be passed to notification listeners and containdetails about a Watch that triggered, such as the watch name or watchtype A notification can provide the following information:

-   1. Watch name (the name of the watch that evaluated to true).-   2. Watch rule type (either harvested or log event rule)-   3. Watch rule (the expression that evaluated to true).-   4. Time watch evaluated to true (formatted, locale specific)-   5. Watch severity level (utilizes same levels as WLS log events)-   6. Watch alarm type (either AutomaticReset or ManualReset)-   7. Watch alarm reset period (for AutomaticReset alarm types)-   8. Watch domain name.-   9. Watch server name.

Watch notifications can also contain payload information about the eventthat triggered the watch. This payload information can be specific tothe type of watch rule. For a log event rule, it can be the attributesof the LogEntry. For an instrumentation data event rule, it can be thedata record items. For a harvester rule, it can be the harvester dataattributes referenced by the rule. The payload information can becontained in a set of key/value pairs. A log watch can provide thefollowing information:

-   1. Id of Log Message.-   2. Name of machine that generated the log message.-   3. Name of server that generated the log message.-   4. Name of thread that generated the log message.-   5. Id of User that generated the log message.-   6. JTA Transaction id of thread that generated the log message.-   7. 18N severity text of log message. See    weblogic.logging.SeverityI18N for values.-   8. Name of subsystem emitting the log message.-   9. Timestamp when the log message was created.-   10. Message content of the log message.-   11. Exception associated with this log event. This is created by    calling the toString method of the Exception.

An instrumentation event watch notification can provide the followinginformation:

-   1. Timestamp when the instrumentation event was created.-   2. Diagnostic Context Id of instrumentation event.-   3. JTA Transaction id of thread that created the instrumentation    event.-   4. Id of User that created the instrumentation event.-   5. Type of instrumentation action.-   6. Name of domain-   7. Name of server that created the instrumentation event.-   8. Name of instrumentation scope-   9. Type of monitor-   10. Source file name-   11. Line number in source file-   12. Class name of join point-   13. Method name of join point.-   14. Method arguments of join point-   15. Return value of join point.-   16. Payload of instrumentation event data.

A harvester watch notification can provide the following information:

-   1. For each attribute referenced in the rule expression, the type of    the MBean, the name of the attribute and the value

Notification listeners are responsible for providing the appropriateimplementation of the particular transportation medium. For example,SMTP notification listeners can provide the actual mechanism toestablish an SMTP connection with a mail server and trigger an e-mailwith the Notification instance that it receives. JMX, SNMP, JMS andother types of listeners can provide their respective implementations aswell. Essentially, any object that wishes to propagate events using aspecific transportation medium can plug into the framework byimplementing the JMX NotificationListener interface accordingly. Thefollowing table describes each notification listener type that WLDFprovides out of the box, as well as the relevant configuration settingsthat correspond to each type of medium:

NOTIFICATION REQUIRED MEDIUM DESCRIPTION CONFIGURATION PARAMETERS SMTPPropagated via regular e-mail. MailSession JNDI name, Destination email,subject (optional, defaulted if not present), body (optional, defaultedif not present) SNMP Propagated via SNMP traps and No configuration isrequired in the watch, but the WLS SNMP Agent SNMPTrapDestination MBeanmust be defined in the WLS SNMP agent. JMX Propagated via standard JMXNone required. Name of Notification is used for runtime notificationsMBean name. JMS Propagated via JMS Message Destination JNDI name.Connection factory JNDI name queues or topics (optional, use default JMSconnection factory if not present). Diagnostic Image Fired notificationinitiates the Location of Image Directory [DIAG-DIMG-FS] notificationprocess of constructing a (optional, use default if not present).Lockout minutes diagnostic image of the server (optional, use default ifnot present).

By default all notifications fired from Watch rules can be stored in theserver log file in addition to being fired through the configuredmedium.

The following notification listener types can also be provided:

-   1. Pager-   2. Node manager restart

The customer can configure a MailSession MBean specifying the SMTP mailsession properties and JNDI name. This can include the various mailproperties of the JavaMail API Specification plus any SMTP mailproperties. In the notification configuration, the customer can specifythe JNDI name of the mail session plus attributes that can not bespecified in the properties file. This can include the destination emailaddress, the subject, and body of the mail message. If the subject isnot configured, then it can default to the following format:

WLS Watch Notification: Watch ‘<watch name>’ with severity ‘<severity>on server ‘<server name>’ has triggered at <date>

If the body is not configured, then it can default to the followingformat:

-   -   WatchName: Name    -   WatchRuleType: Type    -   WatchRule: Expression    -   WatchTime: Time (locale specific format)    -   WatchSeverityLevel: Severity (Same as log event severities)    -   WatchAlarmType: Type (AutomaticReset or ManualReset)    -   WatchAlarmResetPeriod: TimePeriod    -   WatchDomainName: DomainName    -   WatchServerName: ServerName

The body can also have key/value pairs for the event specific payloadinformation. For example, a log event watch would have the followingadditional key value pairs in the body:

-   LogEntryThreadName: ExecuteThread: ‘13’ for queue:    ‘weblogic.kernel.Default (self-tuning)’-   LogEntryLogMessage: The elapsed time of the last snapshot of    diagnostic metrics exceeded the configured sample period by the    following number of milliseconds: 0.-   LogEntryTimestamp: 1077834087283-   LogEntryMachineName: Peter-   LogEntryUserId: system-   LogEntryTransactionId:-   LogEntryId: BEA-320009-   LogEntryServerName: adminServer-   LogEntrySeverity: Warning-   LogEntrySubsystem: Diagnostics-   LogEntryThrowable: null-   WatchDomainName: DiagnosticsTest-   WatchServerName: adminServer

A harvester watch would have the following additional key value pairs inthe body:

HarvesterRuntime.TotalSamplingCycles: 10

The complete list of keys for the log and instrumentation event watchtypes can be found in the weblogic.diagnostics.watch.WatchNotificationclass.

In a server SNMP agent, the customer can configure aSNMPTrapDestionation MBean specifying the SMTP trap destinationconfiguration. In the notification, no SNMP configuration need berequired

If a watch rule evaluates to true and a SNMP trap notification isconfigured, then the server SNMP agent can be used to send the trap toall of the configured SNMP trap destinations. A WebLogic specific trapcan be sent with the specific trap type of 85. The variable bindingname/value pairs for the trap can be as follows.

-   -   trapWatchName—Name of the watch rule that evaluated to true.    -   trapWatchRuleType—Type of watch (either log or harvestable        type).    -   trapWatchRuleExpression—the rule for the watch.    -   trapWatchTime—The time at which the trap was generated        (milliseconds).    -   trapWatchSeverityLevel—The severity of the watch notification        (utilizes same levels as WLS log events)    -   trapWatchAlarmType—If the watch is configured to act as an        alarm, then the type of alarm (either AutomaticReset or Manual).    -   trapWatchAlarmResetPeriod—If AutomaticReset type, then the        amount of time before the alarm is reset.    -   trapServerName—The name of the server on which the trap was        generated.    -   trapWatchData—The set of key/value pairs for the watch specific        payload. This can be in the form key=value,key=value. The keys        are specific to the type of watch. The list of keys can be found        in the weblogic.diagnostics.watch.WatchNotification class.

The customer can configure the JMS subsystem. This includes theconnection factory and JMS destination (queue or topic, distributed ornot). In the notification, the customer can specify the JNDI name of thedestination and optionally the JNDI name of the connection factory.

If a watch rule evaluates to true and a JMS notification is configured,then the listener can create a JMS Key/Value message and send it to theJMS destination using the JNDI name of the destination. If the JNDI nameof the connection factory is not specified, then the JMS defaultconnection factory can be used. The key value pairs in the JMS messagecan be as follows:

Key Value WatchName Name WatchRuleType Type WatchRuleExpressionExpression WatchEvaluationTime Time (formatted, locale specific)WatchSeverity Severity (Same as log event severities) WatchAlarmTypeType (AutomaticReset or ManualReset) WatchAlarmResetPeriod TimePeriodWatchDomainName DomainName WatchServerName ServerName

The JMS message can also contain watch specific payload information asindividual key value pairs in the JMS message.

The customer can specify the notification name for the JMX notification.A runtime MBean can be created during the creation of the notification.The customer can be able to look up the MBean and attach a JMX listenerto the MBean.

If a watch rule evaluates to true and a JMX notification is configured,then the listener can send the Watch Notification to the MBeans. Thecustomer can have added a listener to the specified MBean, and in thelistener can check for the WatchNotification instance, and utilize theinformation in the WatchNotification class to perform the desiredaction.

In the notification configuration, the customer can optionally specifythe diagnostic image directory and lockout minutes.

If a watch rule evaluates to true and an image notification isconfigured, then the listener can call the ImageManager to create thediagnostic image.

Non-Limiting Example of Archive

The Archive component can manage the persistence of collected data andpublished events. Additionally, the Archive provide access to datastored in various log files. The data thus collected cam be persisted ondisk so that it is available for observing trends and root causeanalysis. The data can be collected and persisted in an efficientmanner, without damaging performance significantly. The cost ofcollecting and persisting data must be within reasonable limits. Thepersisted data can be available for rendering and analysis by tools bothwhen the server is running and is shutdown. Also, certain configurationaspects of the archive can be tuned by the user and must be exposed tothe user in a standard manner.

An Archive can logically consist of three components and be instantiatedon a per-server basis: data archive, event archive, and log archive. TheArchive can internally manage the persistence of data collected fromvarious sources as well as published event data that the Archiveconsumes. The collected data can be pushed into the Archive by theHarvester and other data provider components at regular intervalsdefined by the sampling period. On the other hand, event data can bespontaneously generated on the occurrence of some particular event.These events are different from logging events which go to the serverlog file. For example, data coming from a trace event would be publishedto the Archive and thus persisted. The Archive can expose two sets ofinterfaces, one for reading and one for writing. The writing interfacesare exposed to the data providers and data publishers. The readinginterface can be invoked by the Accessor and other data retrievalcomponents to present the persisted data to external clients based ontheir request. The persisted data can be available even when the serveris not running, hence the Archive component may be able to beinstantiated in a non-server process. In one embodiment, when running ina non-server process, only the reader interfaces can be honored. TheArchive can be defined by its Reader and Writer interfaces, thus theactual persistence mechanism can be transparent to all other componentsthat interface with the Archive.

The Writer interfaces can be exposed to data providers and datapublishers. These interfaces can define how the data to be persisted ispresented to the Archive. The Reader interfaces can be exposed to dataretrieval components to access the persisted data of the Archive. Theconfiguration artifact can be responsible for the initialization andconfiguration of the Archive for offline as well as online access modes.

The Writer interfaces can be exposed to data providers and datapublishers so that they may instruct either the sampled or event data tobe persisted. The Writer interfaces can consist of the DataWriter andEventWriter interfaces for persisting sampled data and events,respectively. The Writer interfaces need not be functional when theArchive is running in offline mode.

The DataWriter interface, which can be implemented by the Archivecomponent, can be invoked by the Harvester component of the diagnosticframework or other data provider components. It defines a single methodto persist the sampled data collected over the last scan of allcurrently harvested values. The data sample itself is represented by theDataSample interface.

The EventWriter interface, which can be implemented by the Archivecomponent, is invoked by the event source to persist the generatedevent. It can define a single method to persist the event datarepresented by the EventData interface.

Reader interfaces can be implemented by the Archive component forread-only access to the archived data. The DataReader interface candefine the contract between the Archive and data retrieval components(e.g. the Accessor). Similarly, the EventReader and LogReader interfacescan allow the event and log data to be read in an implementation neutralmanner.

The data reader can allow the Accessor and other data consumptioncomponents to obtain the metadata about the information stored in theArchive, as well as to fetch the data requested based on inputparameters. The event reader allows the event data to be retrieved fromthe Archive. The log reader can allow various log events to be retrievedfrom the Archive.

An Archive configuration artifact can be the central means ofconfiguring and initializing the Archive. The Archive may be manifestedin various forms, such as a flat file or XML database, with each formhaving its own specific set of configuration and initializationparameters. The default native format of the Archive can be a file basedimplementation, however certain tools can be supported to convert fromthe native format to other formats (XML, etc). Configuration parametersfor the file based implementation of the Archive are listed here,however configuration parameters for other internal implementations ofthe Archive are out of the scope of this document.

The archive configuration can be provided by the ArchiveManager artifactthat can capture the locations and size limitations for the persisteddata.

The Archive component can be capable of providing read-only access tothe stored data when present in offline access mode. In online accessmode, both the reader and writer access can be granted. In offline mode,the Archive can utilize MBean stubs for its configuration purposes.

Non-Limiting Example of Harvester

The Harvester can be a central component of the WebLogic DiagnosticFramework that is primarily responsible for harvesting and collectingdata exposed via customer MBeans as well as other harvestable datasources. The Harvester can be capable of balancing dynamism andperformance to harvest diagnostic data in an efficient and accuratemanner while supporting various types of harvestable entities. In thefollowing sections, the important features and functionality of theHarvester are discussed: definition of harvestable entities,registration and unregistration of harvestable entities, and the controlinterfaces that the Harvester exposes for the activation anddeactivation of harvested data.

The most common way of exposing runtime diagnostic data for managedresources in J2EE is to query layered JMX MBeans that express meaningfulstate for various components of resources. However, other sources ofdiagnostic data may also wish to engage in the collection process. Mostharvestable entities expose a significant amount of attributes forresources, and may not always be important or meaningful for the enduser who wishes to collect specific data for various managed resources.The Harvester addresses this issue by incorporating a dynamic harvestconfiguration feature that interacts with the Manager to determine whichsubset of harvestable attributes should be harvested at any given momentin a running WLS instance. The dynamic harvest configuration componentessentially enables the Manager to activate and deactivate thecollection of specific harvestable diagnostic attribute values.

Once data is collected, it must be persisted for further root causeanalysis tools to take control and perform analyses of the system statethat was captured. Therefore, the Harvester must not only interact withthe Manager, but also the Archive component of the diagnostic framework.The interfaces that the Harvester exposes to the Manager, Archive, andother subsystems must be well defined and extensible so that it can beexpanded to possibly tap into other sources of diagnostic resourcesapart from JMX MBeans. Another significant issue regarding the Harvesterinvolves the balance between dynamism and performance. One can imagine ascenario in which multiple subsystems are concurrently accessing theHarvester and instructing it to harvest only certain types of attributesfor specific harvestable entities. The Harvester must be able todynamically reconfigure itself based on requests from the Manager, butat the same time be able to actually harvest the data and transport itto the Archive. Therefore, a mechanism must be introduced to ensuredynamic behavior while still maintaining a system that performs ascustomer requirements dictate.

In one embodiment, there are two ways in which an entity can mark itselfas a harvestable resource for data consumption via the Harvester. Bothmechanisms can provide access to the following information: harvestableattributes, values of harvestable attributes, metadata for harvestableattributes, and finally a name for the harvestable entity. The twomechanisms are differentiated based on the types of harvestableentities. The first manner involves implementing a customer MBean andregistering it as a harvestable entity with the Harvester. The secondapproach allows non-JMX resources to implement the HarvestableDataSourceinterface to participate in the data collection feature of theHarvester. The Harvester can thus be capable of appropriatelyintrospecting the harvestable resources to engage in the collectionprocess.

Once an entity has marked itself as harvestable by implementing theappropriate interface, it can register itself with the Harvester. Thereare two registration/unregistration approaches that are dependent uponthe specific definition of the harvestable entity. If a customer MBeanis to have its attribute values harvested, it invokes the MBeanregistration API of the Harvester. However, if the harvestable entityhas implemented the HarvestableDataSource interface, it must explicitlyregister and unregister itself with the Harvester via the specificmethods defined on the HarvesterRegistrationHandler interface.

The HarvesterManager serves as the primary means of configuring theHarvester dynamically. The HarvesterManager can support the persistenceof Harvester configuration settings across server restarts. As well,system administrators are able to dynamically disable or enable theHarvester while retaining the settings of which harvestable resourcesand attributes are being harvested.

An entity can possess the following primary characteristics in order tobe categorized as harvestable:

-   -   1. List of harvestable attributes    -   2. Harvestable attribute metadata    -   3. Harvestable attribute values    -   4. Registration/Unregistration    -   5. Unique identifier for entity

Customer MBeans that wish to engage in the collection process of theHarvester c can register as such via the HarvesterRegistrationHandler.For MBeans, the Harvester assumes that all attributes are harvestable,metadata is provided via the MBeanInfo/MBeanAttributeInfo objects, thename is the MBean ObjectName, the attribute values are exposed viastandard getter methods, and the registration/unregistration facilitiesare provided via the HarvesterRegistrationHandler interface.

Non-MBean harvestable entities can implement the HarvestableDataSourceinterface to engage in the collection process of the Harvester. Forthose entities that implement HarvestableDataSource, the Harvester caninvoke methods for retrieving the harvestable attribute names,retrieving harvestable attribute values, retrieving harvestableattribute metadata, and retrieving the name of the harvestable datasource. The registration/unregistration facilities can be provided viathe HarvesterRegistrationHandler interface.

The following table expresses the symmetry between the two approachesfor defining a harvestable entity. Note that the primary differencebetween the two approaches is that the MBean approach leveragesfunctionality of JMX that can be utilized to fulfill the harvestablecriteria. However, the HarvestableDataSource approach enforces theimplementer to express these characteristics in a more direct fashion.

Harvestable Criterion Harvestable Customer MBeans HarvestableDataSourceList of harvestable attributes All attributes defined on MBeangetHarvestableAttributeNames are assumed to be harvestable Harvestableattribute metadata MBeanInfo MBeanAttributeInfogetHarvestableAttributeMetadata Harvestable attribute values Exposed viastandard getter getHarvestableAttributeValue methods on the MBeanRegistration/Unregistration addHarvestableMBean addHarvestableDataSourceremoveHarvestableMBean removeHarvestableDataSource Unique identifier forentity MBean ObjectName getName

The Harvester can collect data based on primitive values only, not oncomplex types. The Harvester collects the following primitives:

-   -   1. Integer    -   2. Long    -   3. Double    -   4. Boolean    -   5. String    -   6. Character    -   7. Byte

Harvestable entities can be uniquely identified by combining thefollowing characteristics:

-   -   1. Unique name    -   2. Unique type    -   3. Server    -   4. Domain

The above characteristics are encapsulated in implementations of theDiagnosticObjectName interface.

Customer MBeans that wish to participate in the collection process ofthe Harvester can explicitly register as such via a well-definedregistration interface. Harvestable customer MBeans are encouraged toexplicitly unregister with the Harvester, however the Harvester candetect MBeans that no longer exist and remove them from the list ofMBeans for data collection.

The Harvester can also consume harvestable data from non-JMX resources.For example, diagnostic metric data published from other components inthe diagnostic framework as well as other potential sources ofinformation (such as SNMP) can also be harvested via a pluggablemechanism. Essentially, the non-JMX harvestable data source can providean implementation of the HarvestableDataSource interface so that theHarvester can extract the necessary information to perform collection onbehalf of the harvestable data source. Since the HarvestableDataSourcelives outside of JMX, there can be an explicit manner in which the datasource can register and unregister itself with the Harvester.

Steps for a developer to register an MBean and mark it as a harvestingcandidate can be:

-   -   1. Developer writes an MBean.    -   2. Developer registers MBean with the MBean Server.    -   4. Developer registers MBean with the Harvester via        addHarvestableMBean( ).    -   5. Harvester is notified of the newly registered MBean and        records it internally.

Steps for Registering Other Harvestable Data Sources for the Harvester

-   -   1. Developer writes an object.    -   2. Developer implements the HarvestableDataSource interface.    -   3. Developer registers the harvestable entity with the Harvester        via addHarvestableDataSource( ).    -   4. Harvester is notified of the newly registered entity and        records it internally.

Note that for both registration approaches, the Harvester can scheduleany newly registered harvestable entities for the next sampling pass ifthe entities are registered while the Harvester is currently collectingmetrics for existing harvestable entities.

The ObjectName format generated for ServletRuntime MBeans follows adynamic naming convention that appends a number at the end of theservlet's name. This number corresponds to the actual instance of theservlet in memory, as a single servlet type may be instantiated a numberof times in a WLS instance. The Harvester can be aware of the dynamicnature of each servlet's name in order to aggregate the metrics of eachinstantiated ServletRuntime MBean per servlet name. For example, acustom servlet entitled MyServlet may be instantiated ten times in aserver instance, which results in ten unique ServletRuntime MBeaninstances. As these instances are registered in the MBean Server, theHarvester must be able to recognize that each instance is related viathe name “MyServlet” and begin harvesting metrics for the servletinstances immediately (assuming that a client has instructed theHarvester to collect metrics for a particular servlet). Note that theconsole can display all instances of ServletRuntime for a particularservlet as “rolled up” into a single item such as “MyServlet”, insteadof listing all of the instances with dynamically generated numbers as aharvesting selection point. The following listings and table describethe actions that the Harvester takes for harvesting a regular MBean aswell as the special ServletRuntime case:

Instantiated JDBCConnectionPoolRuntime MBean

-   -   1. Harvester receives notification of instantiated instance.    -   2. If the MBean type has been previously recorded for        harvesting, then record this instance for harvesting.    -   3. If the MBean type has not been previously recorded for        harvesting, then record this instance and type as harvestable.        Instantiated ServletRuntime MBean    -   1. Harvester receives notification of instantiated instance.    -   2. If the MBean type has been previously recorded for        harvesting, strip off the generated number at the end of the        servlet's name and determine if that particular servlet has been        recorded for harvesting.    -   3. If the specific servlet has been scheduled for harvesting,        then record the newly instantiated ServletRuntime MBean for        harvesting.    -   4. If the MBean type has not been previously recorded for        harvesting, then record this instance and type as harvestable.        Regular/Irregular MBean Harvesting Characteristics Table

Characteristic JDBCConnectionPoolRuntime ServletRuntime Harvestable typeJDBCConnectionPoolRuntimeMBean ServletRuntimeMBean Harvestable instanceObjectName ObjectName name Selection of attribute toJDBCConnectionPoolRuntime: ServletRuntime: be harvested for typeactiveConnectionsCurrentCount invocationTotalCount Selection of instanceto Original ObjectName Original ObjectName with trailing be harvestedfor type number stripped

The Harvester exposes its configuration interface via a centralHarvesterManager. The HarvesterManager can initially allow operations tobe performed on harvestable types as opposed to instances. Systemadministrators and other users can invoke this management interface todo the following:

-   -   1. Retrieve all harvestable types.    -   2. Set the Harvester's configured sampling period (same for all        harvestable types).    -   3. Configure the Harvester to collect data for specific        attributes.    -   4. Configure the Harvester to not collect data for specific        attributes.    -   5. Retrieve all currently harvested attribute names for a        particular harvestable type.    -   6. Retrieve all currently harvested attribute values for a        particular harvestable instance.    -   7. Completely enable or disable the Harvester while retaining        user's settings.

Harvestable entities can be represented as a specific type (e.g.:JDBCConnectionPoolRuntimeMBean) and instances of that particular type.The Harvester allows configuration at the following levels:

-   -   1. Configuration of attributes to be harvested at the “type”        level only. All selected attributes to be harvested can be        applied to all instances of the type.    -   2. Configuration of which specific instances of a particular        type can be candidates for harvesting.

Note that the HarvesterManager API can supports adding/removingattributes at the type level only. For example, if for a particularMBean type there are five instances configured for harvesting, thenadding or removing harvested attributes affects all of the instances ofthe specified MBean type. If no instances of an MBean are specified, butrather only an MBean type, then the Harvester configures the attributesto be harvested for the type and applies it to any future MBeaninstances of that type that register with the Harvester.

Once the user has set the sample period for the Harvester via theHarvesterManager, the following actions occur:

-   -   1. Harvester polls/samples the harvestable entities (at unified        sample period) and collects the latest values for currently        harvested attributes.    -   2. Read-through MBeans (those that only update their values when        the respective getAttributeName( ) method is called) are handled        properly.    -   3. The time at which harvested attributes are collected from the        harvestable entities are bundled with the sample as a “time        stamp.”    -   5. The Harvester pushes the collected diagnostic data (those        attributes to be persisted) to the Archive.

Note that the Harvester can operate in two modes: it can harvestattributes and not persist them to the Archive, and it can also harvestattributes and persist them to the Archive. Once the data has beenharvested and is ready to be propagated to the Archive, the Harvestercan attempt to provide various statistics that the developer or systemadministrator may find useful when observing the collected diagnosticdata, such as:

-   -   1. How long it takes to retrieve the names of all of the        harvestable attributes for a harvestable entity.    -   2. How long the getAttributeName( ) (where “AttributeName” is a        valid name of an attribute) takes to return with a value.    -   3. Scan/Sample time (how long it takes the Harvester to find and        invoke an attribute value method).    -   4. Number of attributes harvested.

Other relevant statistics may be collected as well and incorporated intothe harvested diagnostic data. Additionally, an error entry can beplaced in the log file if the time it takes to sample a harvestableattribute exceeds the sample period.

The Harvester can maintain various internal house-keeping datastructures to record which harvestable entities and attributes arecurrently being harvested. System administrators invoking theHarvesterManager directly affect the behavior of the Harvester, asoutlined in the previous section. The HarvesterManager can also providepersistence capabilities so that subsequent server restarts can retainthe user's harvesting settings. Furthermore, in order to support theenabling and disabling of the Harvester at runtime, the user's settingsmust also be recorded within the Harvester and persisted if the userwishes to retain those settings for a future server instance.

The HarvestableDataSource implementations can be automatically generatedfor server runtime entities that expose diagnostic metrics. The runtimeartifacts can be marked/tagged with javadoc tags that can besubsequently processed to generate the HarvestableDataSourceimplementations. The code generation tool can extract annotations toprovide: list of harvestable attribute names, harvestable attributevalues, harvestable attribute metadata, as well as the name of theharvestable data source. Note that there can also potentially be aninternal static harvestable list for server runtime entities in order toavoid any overhead of explicitly registering them with the Harvesterupon server startup.

Logging Framework

The Logging framework can perform efficiently with a low overhead and weneed to identify possibilities where we can make Logging faster andscalable for increased throughput. The log content should also includeadditional context information to support easy analysis by allowingfiltering of messages emitted within the scope of a particularapplication request. The J2EE Servlet APIs provide an interface for WebApplications to log application related events using the ServletContext.These messages can go to the webapp specific log as opposed to theserver log where it gets directed today. Similar logging facility can beprovided to a J2EE Resource Adaptor for ManagedConnectionFactory andManagedConnection scoped logging.

Enhancements to the LogMBean interfaces allow users to control thelogging output by means of levels and filters.

Non-Limiting Example of Diagnostic Manager

The Diagnostic Manager can allow for the configuration and control ofcomponents in the diagnostic framework. It can define the diagnosticframework configuration API for Administrative clients such as theWebLogic Administration Console, command line tools, scripting utilitiesor other third-party tools to configure and control the behavior of theDiagnostic Framework components. The Manager can also defines theruntime interfaces that describe the state of the Diagnostic Framework.

The Diagnostic Manager can be a logical component that is made up of twosets of interfaces, a Configuration interface for configuring theruntime behavior and a Runtime interface for accessing the runtime stateof the diagnostic framework. The Configuration interfaces can be used toconfigure and control the different components of the DiagnosticFramework. The configuration can take place on the Admin server and thechanges can be pushed to the Managed Server, where the updates can beapplied. The Runtime interfaces can reflect the current state and alsoexpose operations on the different components of the DiagnosticFramework. Both the Configuration and the Runtime interfaces can beexposed as MBeans.

It is assumed that OA&M Configuration can provide a standard mechanismof configuration persistence and distribution to the managed servers inthe server domain. The Diagnostic Framework can make use of the OA&Minfrastructure for this purpose. The Diagnostic Manager need not provideany mechanism to view or retrieve the data collected by the Harvestercomponent. This can be provided by the Accessor component of theDiagnostic Framework.

The diagnostic configuration can define Configuration Beans that can beused to configure the WLDF.

The Runtime interfaces of the Manager can expose the runtime state andoperations defined for the different components of the diagnosticframework.

The diagnostic framework configuration can have two aspects, elementsthat are defined and configured at the Server level and elements thatare at defined at the domain level and deployed to a Server. TheServerDiagnosticMBean can contain the configuration elements of thediagnostic framework applicable to each WebLogic Server instance. TheServerDiagnosticMBean can be a child of the ServerMBean. TheDiagnosticDeploymentMBean can be defined at the domain level andtargeted to individual servers. This can enable multiple servers toshare the configuration aspects that are defined at the domain level. Amanaged server is however associated with at most one instance of theDiagnosticDeploymentMBean.

The ServerDiagnosticMBean can be a child of the ServerMBean and candefine configuration aspects of the diagnostic framework that aredefined at the Server scope. This includes the directory location forthe image sources generated by the diagnostic framework.

The DiagnosticDeploymentMBean can define the configuration aspects ofthe diagnostic framework that can be defined at the domain level andtargeted to individual servers. It can contain the configurationelements for the Instrumentation, Harvester and WatchNotificationcomponents of the diagnostic framework.

The Instrumentation MBean can interact with the InstrumentationFramework and enable selected monitors with configured advices at thedefined point cuts. The Instrumentation MBean can enable the followinguse case scenarios through the APIs that it exposes.

Instrumentation Configuration Primitives of One Embodiment:

-   -   The user enables or disables the entire instrumentation behavior        in the given scope, which could be either application or sever.    -   The user specifies what classes needs to be included in the        instrumentation in a given scope.    -   The user excludes certain classes from getting instrumented by        the different monitors in this scope.    -   The user adds/removes a monitor of a specified class to the        instrumentation scope. They also select the point cuts for the        monitors which could be fixed for standard and delegating        monitors.    -   The user specifies the configuration properties of a given        monitor.    -   The user enables or disables a specific monitor.    -   The user adds/removes an advice from a given monitor.    -   The user specifies the configuration properties of an advice.

The Harvester MBean can define how the diagnostic data is collected bythe Harvester component of the diagnostic framework. The Harvester MBeancan support the following configuration primitive operations that can bepart of the harvester configuration scenarios. Harvester configurationprimitives of one embodiment:

-   -   The user enables or disables the Harvester component, so that        data collection and persistence is either enabled or disabled        for the harvested attributes.    -   The user sets a new value for the global sampling rate for the        harvester.    -   The user queries the manager to show all the known harvestable        types known to it.    -   The user queries the manager for all the known harvestable        attributes for a given type.    -   The user selects a harvestable type and specifies the list of        attributes that should be enabled for collection for the given        type.    -   The user queries the manager for all the currently harvested        types.    -   The user queries the manager to show the currently harvested        attributes for a harvestable type.    -   The user adds/removes attributes from the set of currently        harvested attributes for a specific type.    -   The user queries the manager for all the known harvestable        instances for a given type.    -   The user specifies the set of instances to be harvested for the        specified type.    -   The user adds/removes an instance from the harvested set for the        specified type.

The Watch Notification MBean can define how watches are selected, rulesdefined and notifications issued to the specified destinations document.

Watch and Notification Configuration Primitives of One Embodiment:

-   -   The user enables or disables all the configured watches        globally.    -   The user creates a new Watch and selects the instances to be        watched, along with their attributes. Only instances that are        currently being harvested can be added to the Watch.    -   The user constructs the rule for the watch based on the        attributes of the selected instances.    -   The user deletes a specific Watch. Note that this can also        delete the associated rule definition.    -   The user adds different types of notifications that can be        emitted when a watch rule is triggered. Note that the event        causing the watch rule trigger is always persisted in the event        log.    -   The user removes all or some of the notifications emitted when a        watch rule is triggered. When no notification is enabled, the        event can be persisted in the event log.    -   The user creates a watch to observe log events specifying the        filtering rules for the log watch.        The runtime interfaces define the runtime state of the different        diagnostic framework components. They also expose the operation        defined for each component. The instances of these interfaces        can live on the individual managed servers. These interfaces are        defined as Runtime MBeans so that a JMX client can easily access        them. The DiagnosticRuntime interface can encapsulate all other        runtime interfaces for the individual diagnostic framework        components.

Non-Limiting Example of Accessor Component

The Weblogic Diagnostic Framework can capture a variety of diagnosticdata from various sources. The Accessor component can provide access tothis data to the internal and external clients such as console, thirdparty tools etc.

The Diagnostic Framework can capture and maintains a variety ofdiagnostic data from various sources in a running server. Suchinformation includes data in runtime MBeans, non-jmx data sources,various log files, trace and event files, etc. Thus, diagnosticframework acts as a central repository of diagnostic information. Thisinformation can be used by human operators as well as tools to identifyproblems in a production environment. In order for this information tobe useful, it needs to be made accessible to suitable clients. Theaccessor component of diagnostic framework provides a single accesspoint to clients. Such clients can be internal tools such as Console, orthey can also be external third party tools. The accessor can providefunctionality to supply data to clients on demand.

The diagnostic framework maintains diagnostic information per server.Consequently, the accessor component of diagnostic framework on a serverprovides access to diagnostic information for that server. Thediagnostic framework can maintain different kind of diagnosticinformation in separate data stores, which can be modeled as tabulardata. Using the accessor APIs, clients can access diagnostic data intabular form.

The accessor functionality, which can be available from a running servercan be exposed as a runtime MBean.

Historical diagnostic data, such as archived metrics, log and eventfiles, etc can be available even when the server is not running. Theaccessor component can enable offline tools to access such data byproviding suitable functionality to offline tools. Such offline accesscan be possible only from the local server.

The accessor can provide a “synchronous pull” model for the clients toaccess diagnostic data from diagnostic framework. It need not coversending data to interested clients asynchronously. This document alsodescribes functionality which can be used to build offline tools toaccess historical diagnostic information when the server is not running.

The accessor can provide metadata on what data is available, such as logfiles, harvested metrics and historical data on a server. The accessorcan allow clients to access data captured by the diagnostic framework,which includes harvested metrics, logs and events. Accessor componentcan make part of its functionality available to offline tools to accesspersisted diagnostic data such as log files and historical harvesteddata. Thus, persisted diagnostic data can be made accessible even when aserver instance is not running. Offline access can be available onlylocally. Access to historical data in offline mode can rely on filesystem protection mechanisms.

The accessor can provide unified access to diagnostic data in disparatedata stores. Although different data-stores may have different layoutsin terms of the attributes (columns), they can be modeled as tabulardata. The accessor can provide a query language to query data from thedata stores.

Often, it is more efficient to stream large amount of data to remoteclients, rather than require them to access data in piece-meal manner.This can be achieved by providing an access servlet to remote clients toaccess certain diagnostic data. Data can be streamed into the servlet'soutput stream, which can be read by the remote client. For example, withthis mechanism, it can be possible to efficiently transmit raw data fromsections of log files. Similarly, a large amount of collected diagnosticmetric and event data can be streamed out as an XML document, which canbe consumed by the remote client.

The diagnostic framework can collect diagnostic information fromdifferent sources. The consumers of this information should know whatdiagnostic information is collected and available. The accessorcomponent can provide functionality to programmatically obtain such Metainformation.

The accessor component can provide information regarding availablediagnostic information from other diagnostic framework components.Information captured by diagnostic framework can be segregated intological data stores, separated by the types of diagnostic data. Forexample, Server logs, HTTP logs, JDBC logs, harvested metrics etc, canbe captured in separate data stores. The accessor component can providefunctionality to discover such data stores and find out about the natureof data they contain.

Data stores can be logically viewed as a collection of tabular data.Each record in the table represents a datum. Columns in the tabledescribe various characteristics of the datum. Different data stores mayhave different columns. However, most data stores can have some sharedcolumns, such as the time when the data item was collected.

The Meta information made available can include:

-   -   List of supported data store types, e.g., SERVER_LOG, HTTP_LOG,        HARVESTED_DATA etc    -   List of available data stores.    -   Layout of each data store, i.e., information about columns in        the data store.

Exceptional situations, such as request for information on anon-existent data store can cause appropriate exception to be thrown atthe caller, as described in the interface section below.

The accessor can make the Meta information available through followinginterfaces. DiagnosticCurrentAccessService interface contains methods toidentify supported data-store types as well as names of availabledata-stores. It also provides a method to get a reference to a nameddata-store represented by an instance implementingDiagnosticDataAccessService interface. The DiagnosticDataAccessServiceprovides a number of methods to get Meta information about the datastore.

The accessor can provide functionality to access data captured indifferent data-stores. Although each data-store may have differentlayout, data can be accessed using the single unified API. Data isqueried by specifying a query expression in string form. All recordsfrom the data-store which match the specified query can be returned. Byspecifying appropriate constraints in the query, a client can filter thedata in different ways.

Although different data-stores may have different layouts, the accessorcan provide a unified access mechanism for all data stores.

The accessor can provide functionality to selectively access data fromdifferent data stores. Data to be accessed can be selected based onspecified query criterion. Based on the query, a client can limit thevolume of data sent to it.

Exceptional situations can cause an InvalidQueryExpression exceptionthrown to the called. The conditions can include:

-   -   Syntax error in the query expression    -   Invalid column names in the query    -   Semantic errors in the query, such as semantically inconsistent        operations with columns in the data store

Persisted diagnostic data such as historical harvested data and logfiles provide a rich source of information for identifying problems. Itcan be accessible even when a server instance is not running. Asdescribed in previous sections, the accessor component can make thisinformation available to offline tools through defined interfaces. Theaccessor component provides the underlying functionality to access data.The presentation of the data can to be handled by offline tools.

The accessor can provide functionality to selectively access diagnosticdata by specifying a query criterion. The query can be specified as astring, in the query language defined by the accessor. The querylanguage syntax can be a small and simplified subset of SQL syntax. Itspurpose can be to specify the selection criteria to build Booleanexpressions based on the column names supported by the data store. Usingthe query language, it can be possible to specify relatively complexselection expressions which typically appear in the WHERE clause of anSQL SELECT statement.

The query can be executed against a specific data store. Thus, the“FROM” part of the query can be implicit. Upon match, all columns ofmatching rows can be returned. Following example shows the query tocapture harvested attributes of JDBC connection pool ‘MyPool’ within ainterval between start_time (inclusive) and end_time (exclusive).

( (TIMESTAMP >= start_time AND TIMESTAMP < end_time)  AND  (TYPE =‘JDBCConnectionPoolRuntimeMBean’ AND  INSTANCE_NAME = ‘MyPool’) )

The variables in the criterion expression can be the column names of thedata store. If a column is of Boolean type, it can be used directly inthe expression. Otherwise, it can be used in a comparison operationconsistent with its type.

Literals can participate in comparison operations. Literals can benumeric, or string types. Numeric literals can be integers or floatingpoint numbers.

String literals are enclosed with double quotes. A percent (‘%’)character is used as a wild card inside string literals. A backslash(‘\’) character is used to escape special characters, (such as doublequotes and percent characters) in string literals if needed.

The query language can allow binary comparison of data-store columnswith literals. Thus, expressions of the type (variable OP literal) canbe possible. Supported binary comparison operators can include:

Operator Operand Type Example Description = Numeric, String VALUE = 100True if specified column value equals 100 <> Numeric, String VALUE <>100 True if specified column value does not equal 100 > Numeric VALUE >100 True if specified column value is greater than 100 >= NumericVALUE >= 100 True if specified column value is greater or equals to 100< Numeric VALUE < 100 True if specified column value is less than the100 <= Numeric VALUE <= 100 True if specified column value is less orequals to 100 like String TYPE like “% Runtime” True if specified columnvalue ends with ‘Runtime’ like String TYPE like “Web %” True ifspecified column value starts with ‘Web’ like String TYPE like “%Component %” True if specified column value contains ‘Component’

Complex query (sub) expressions can be built using subexpressionsinvolving variables, binary comparisons and other complexsubexpressions.

-   -   a AND b will evaluate to true, iff both subexpressions a and b        evaluate to true.    -   a OR b will evaluate to true iff either subexpressions a or b        evaluate to true.    -   NOT a will evaluate to true iff subexpression a evaluates to        false.    -   (a) will evaluate to true iff subexpression a evaluates to true.

Operator precedence in complex expressions can be as follows (fromhighest to lowest):

-   -   ( )    -   NOT    -   =, < >, >, >=, <, <=, like    -   AND    -   OR

One embodiment of the present invention may be implemented using aconventional general purpose or a specialized digital computer ormicroprocessor(s) programmed according to the teachings of the presentdisclosure, as will be apparent to those skilled in the computer art.Appropriate software coding can readily be prepared by skilledprogrammers based on the teachings of the present disclosure, as will beapparent to those skilled in the software art. The invention may also beimplemented by the preparation of integrated circuits or byinterconnecting an appropriate network of conventional componentcircuits, as will be readily apparent to those skilled in the art.

One embodiment includes a computer program product which is a storagemedium (media) having instructions stored thereon/in which can be usedto program a computer to perform any of the features presented herein.The storage medium can include, but is not limited to, any type of diskincluding floppy disks, optical discs, DVD, CD-ROMs, micro drive, andmagneto-optical disks, ROMs, RAMs, EPROM's, EEPROM's, DRAM's, flashmemory devices, magnetic or optical cards, Nanosystems (includingmolecular memory ICs), or any type of media or device suitable forstoring instructions and/or data.

Stored on any one of the computer readable medium (media), the presentinvention includes software for controlling both the hardware of thegeneral purpose/specialized computer or microprocessor, and for enablingthe computer or microprocessor to interact with a human user or othermechanism utilizing the results of the present invention. Such softwaremay include, but is not limited to, device drivers, operating systems,execution environments/containers, and user applications.

The foregoing description of preferred embodiments of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Many modifications andvariations will be apparent to one of ordinary skill in the relevantarts. For example, steps performed in the embodiments of the inventiondisclosed can be performed in alternate orders, certain steps can beomitted, and additional steps can be added. The embodiments were chosenand described in order to best explain the principles of the inventionand its practical application, thereby enabling others skilled in theart to understand the invention for various embodiments and with variousmodifications that are suited to the particular use contemplated. It isintended that the scope of the invention be defined by the claims andtheir equivalents.

1. A method comprising: instrumenting classes of server software with adiagnostic monitor, the diagnostic monitor including at least onediagnostic function; and enabling the diagnostic function at thediagnostic monitor while the server software is running withoutrestarting the server software; wherein a function is added or removedfrom the diagnostic monitor by adding or removing an advice from a listof advices for the diagnostic monitor.
 2. The method of claim 1, whereinthe diagnostic function is a counter.
 3. The method of claim 1, whereinthe diagnostic function includes saving arguments.
 4. The method ofclaim 1, wherein the diagnostic function are limited to a selectedpredetermined set of actions.
 5. The method of claim 1, wherein theinstrumenting includes weaving an indication of the diagnostic monitor.6. The method of claim 1, wherein a library of diagnostic monitors isprovided.
 7. The method of claim 1, wherein a different diagnosticfunction can be selected for the diagnostic monitor while the serversoftware is running without restarting the server software.
 8. Themethod of claim 1, wherein each diagnostic monitor is associated with apointcut.
 9. A computer readable storage medium including codecomprising: an instrumentation unit to instrument server software at apointcut with a diagnostic monitor, and the diagnostic monitor having atleast one diagnostic function, wherein diagnostic actions can be enabledfor a pointcut at the diagnostic monitor while the server software isrunning without restarting the server software; wherein a function isadded or removed from the diagnostic monitor by adding or removing anadvice from a list of advices for the diagnostic monitor.
 10. Thecomputer readable storage medium of claim 9, wherein the diagnosticfunction is a counter.
 11. The computer readable storage medium of claim9, wherein the diagnostic function includes saving arguments.
 12. Thecomputer readable storage medium of claim 9, wherein the diagnosticfunction are limited to a selected predetermined set of actions.
 13. Thecomputer readable storage medium of claim 9, wherein the instrumentingincludes weaving in the diagnostic monitor.
 14. The computer readablestorage medium of claim 9, wherein a library of diagnostic monitors isprovided.
 15. The computer readable storage medium of claim 9, whereindifferent diagnostic function can be selected for the diagnostic monitorwhile the server software is running without restarting the serversoftware.
 16. The computer readable storage medium of claim 9, whereineach diagnostic monitor is associated with a pointcut.
 17. A methodcomprising: instrumenting classes of server software with a diagnosticmonitor; and selecting and enabling a diagnostic function for thediagnostic monitor while the server software is running withoutrestarting the server software; wherein a function is added or removedfrom the diagnostic monitor by adding or removing an advice from a listof advices for the diagnostic monitor.
 18. The method of claim 17,wherein the diagnostic function is limited to a selected predeterminedset of actions.
 19. The method of claim 17, wherein the instrumentingincludes weaving an indication of the diagnostic monitor.
 20. The methodof claim 17, wherein a library of diagnostic monitors is provided. 21.The method of claim 17, wherein each diagnostic monitor is associatedwith a pointcut.
 22. A method comprising: instrumenting classes ofserver software with a diagnostic monitor, the diagnostic monitorincluding at least one diagnostic function; and enabling the diagnosticfunction at the diagnostic monitor while the server software is runningwithout restarting the server software; wherein a dye mask is used atthe diagnostic monitor so that the diagnostic function is only run for asubset of requests based on the dye mask and a dye vector of therequest.